Help with HostHatch private network, can't see the other side

tarasis · December 2022

Struggling to get this running, and was hoping someone might be able to point me in the right direction.

I have enabled Private networking on old and new storage servers, and power cycled as instructed.

Old server is running Arch Linux, and uses netctl for networking.
New server currently Ubuntu. (will probably swap to Arch, want to get this working first).

Running ip -o link | grep 00:22 on both devices shows an ens4 device.

3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000\    link/ether 00:22:snip brd ff:ff:ff:ff:ff:ff\    altname enp0s4

On the Arch box, I have added an internal-network file with

Interface=ens4
Connection=ethernet
IP=static
Address=('192.168.10.3/24')
MACAddress=00:22:snip
// tried with and without the MACAddress line

On Ubuntu I have /etc/netplan/90-private.yaml per the docs https://docs.hosthatch.com/networking/#private-networking

network:
    ethernets:
        ens4:
            addresses:
            - 192.168.10.2/24
            match:
                macaddress: 00:22<skip>
    version: 2

After enabling respective files, ip link returns showing ens4 is up.

Arch:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:snip brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:22:snip brd ff:ff:ff:ff:ff:ff
    altname enp0s4

Ubuntu:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:20:snip brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:22:snip brd ff:ff:ff:ff:ff:ff
    altname enp0s4

And the routing

Arch:

default via 31.snip dev ens3 
31.snip/24 dev ens3 proto kernel scope link src 31.snip 
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.3

Ubuntu

default via 31.snip dev ens3 
31.snip/24 dev ens3 proto kernel scope link src 31.snip 
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.2

But when I try and ping from one machine to the other I get destination unreachable.

What am I missing? Coz I'm baffled right now

skorous · December 2022

Dumb question but since you don't say, they're at the same location, right?

tcpdump the interface on the destination side and see if you see the traffic coming in?

tarasis · December 2022

@skorous said:
Dumb question but since you don't say, they're at the same location, right?

Good point, I forgot to specify they are both in Amsterdam. Although one is marked "Legacy".

tcpdump the interface on the destination side and see if you see the traffic coming in?

Nothing appears when pinging from the Arch side to the Ubuntu side. The lines below are from me starting a ping on the Ubuntu side

root@backup2tb:~# tcpdump -i 2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
01:35:51.568578 ARP, Request who-has 192.168.10.3 tell backup2tb, length 28
01:35:52.580830 ARP, Request who-has 192.168.10.3 tell backup2tb, length 28

AuroraZero · December 2022

Routing try ens4 not ens3

FrankZ · December 2022

I don't know Arch but it does not look like you have a route set for the private networking. So maybe this might help.
Ubuntu
ip route add 192.168.10.0/24 via 192.168.10.2 dev ens4

Arch:
ip route add 192.168.10.0/24 via 192.168.10.3 dev ens4

skorous · December 2022

@AuroraZero said:
Routing try ens4 not ens3

But the private addresses are on ens4 ... won't that just break his external network?

skorous · December 2022

@FrankZ said:
I don't know Arch but it does not look like you have a route set for the private networking. So maybe this might help.

I have a feeling I'm missing something obvious but isn't that the highlighted lines?

@tarasis said: Arch:

default via 31.snip dev ens3
31.snip/24 dev ens3 proto kernel scope link src 31.snip
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.3

Ubuntu

default via 31.snip dev ens3
31.snip/24 dev ens3 proto kernel scope link src 31.snip
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.2

skorous · December 2022

@tarasis said:

tcpdump the interface on the destination side and see if you see the traffic coming in?

Nothing appears when pinging from the Arch side to the Ubuntu side. The lines below are from me starting a ping on the Ubuntu side
root@backup2tb:~# tcpdump -i 2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
01:35:51.568578 ARP, Request who-has 192.168.10.3 tell backup2tb, length 28
01:35:52.580830 ARP, Request who-has 192.168.10.3 tell backup2tb, length 28

When you say 'starting a ping on the Ubuntu side' you mean you have the tcpdump running on Ubuntu side and have Ubuntu try to ping Arch, right? So the Ubuntu machine is seeing it's own traffic ( guessing so since it's knows its name )? What happens if you tcpdump the Arch side and try the same?

Shouldn't it be tcpdump -i ens4? I've never used a number like that...

FrankZ · December 2022

@skorous said:

@FrankZ said:
I don't know Arch but it does not look like you have a route set for the private networking. So maybe this might help.

I have a feeling I'm missing something obvious but isn't that the highlighted lines?

@tarasis said: Arch:

default via 31.snip dev ens3
31.snip/24 dev ens3 proto kernel scope link src 31.snip
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.3

Ubuntu

default via 31.snip dev ens3
31.snip/24 dev ens3 proto kernel scope link src 31.snip
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.2

Although I am not any kind of routing expert, in my opinion, after the ip route commands I showed above are entered there should be additional route lines
192.168.10.0/24 via 192.168.10.2 dev ens4 - on Ubuntu
192.168.10.0/24 via 192.168.10.3 dev ens4 - on Arch

Because right now I see everything going over the default route.
IIAC The lines you highlighted above show that the related IPs are UP and attached to dev ens4 the on the local machine, but not any routing.

skorous · December 2022

@FrankZ said:

@skorous said:

@FrankZ said:
I don't know Arch but it does not look like you have a route set for the private networking. So maybe this might help.

I have a feeling I'm missing something obvious but isn't that the highlighted lines?

@tarasis said: Arch:

default via 31.snip dev ens3
31.snip/24 dev ens3 proto kernel scope link src 31.snip
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.3

Ubuntu

default via 31.snip dev ens3
31.snip/24 dev ens3 proto kernel scope link src 31.snip
192.168.10.0/24 dev ens4 proto kernel scope link src 192.168.10.2

Although I am not any kind of routing expert, in my opinion, after the ip route commands I showed above are entered there should be additional route lines
192.168.10.0/24 via 192.168.10.2 dev ens4 - on Ubuntu
192.168.10.0/24 via 192.168.10.3 dev ens4 - on Arch

Because right now I see everything going over the default route.
IIAC The lines you highlighted above show that the related IPs are UP and attached to dev ens4 the on the local machine, but not any routing.

They don't need any routing because the kernel knows what network it's on. You only need that if you've got access to something else via that network.

[root@cerebro ~]# ip addr show ens4
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
link/ether 00:22:da:df:41:cf brd ff:ff:ff:ff:ff:ff
altname enp0s4
altname ens4
inet 10.174.2.24/8 brd 10.255.255.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever

[root@cerebro ~]# ip route
default via 193.29.62.1 dev eth0 proto static metric 100

10.0.0.0/8 dev eth1 proto kernel scope link src 10.174.2.24 metric 101

10.89.0.0/24 dev cni-podman1 proto kernel scope link src 10.89.0.1

192.168.0.0/16 dev nebula1 proto kernel scope link src 192.168.2.157 mtu 1400

192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

193.29.62.0/24 dev eth0 proto kernel scope link src metric 100

Sorry about the formatting. Stupid thing.

FrankZ · December 2022

I'll just crawl back in my hole then.

skorous · December 2022

@FrankZ said:
I'll just crawl back in my hole then.

No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

AuroraZero · December 2022

Edit: told you it was stupid

Try putting in the real Mac addresses

skorous · December 2022

@AuroraZero said:
Stupid question did you restart networking on both machines?

Ah, good question.

My best guess is that they each accidentally got their own private network so they don't see each other. Thinking was the tcpdump's would show no traffic either way and since ARPs are broadcast no matter how buggered the OS setup was it should've seen it. But again, this isn't my area either.

FrankZ · December 2022

@skorous said:

@FrankZ said:
I'll just crawl back in my hole then.

No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

AuroraZero · December 2022

@skorous said:

@AuroraZero said:
Stupid question did you restart networking on both machines?

Ah, good question.

My best guess is that they each accidentally got their own private network so they don't see each other. Thinking was the tcpdump's would show no traffic either way and since ARPs are broadcast no matter how buggered the OS setup was it should've seen it. But again, this isn't my area either.

Normally it would but I don't know Hosthatch's setup on these. I was reading their article on it and it hit me but now I think he might not have the right MACs.

skorous · December 2022

@FrankZ said:

@skorous said:

@FrankZ said:
I'll just crawl back in my hole then.

No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

phew Okay good. I feel better. Yeah, I'm a RHEL guy too. Was actually certified back on 6.

The one that has me curious is that he has a mix of old and new names on the Ubuntu box. I was under the impression it was all or nothing although I guess maybe netplan is mapping the name somehow.

AuroraZero · December 2022

@FrankZ said:

@skorous said:

@FrankZ said:
I'll just crawl back in my hole then.

No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

Don't feel bad I work in Slackware. If it was Slack I would have his back. Yes I know it rhymes and is a pun as well. I don't utilize much private networking at least not this type of thing

skorous · December 2022

@AuroraZero said:

@FrankZ said:

@skorous said:

@FrankZ said:
I'll just crawl back in my hole then.

No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

Don't feel bad I work in Slackware. If it was Slack I would have his back. Yes I know it rhymes and is a pun as well. I don't utilize much private networking at least not this type of thing

Started on Slackware back in .... 93'ish? Couldn't have been much later. Dropped out by 95 I think... Still have my 0.99 3.5 inch floppies though I'm sure they've got rampant bitrot by now ...

EDIT: rats, time for bed. Was just having fun too. >:(

AuroraZero · December 2022

@skorous said:

@AuroraZero said:

@FrankZ said:

@skorous said:

@FrankZ said:
I'll just crawl back in my hole then.

No, no ... you should definitely check my work. Sorry if that sounded bad. I was as much as anything else saying it to make sure I wasn't wrong.

I didn't take it badly, I just do not consider myself a routing expert and I work with RH based O/S so since this is Ubuntu and Arch I am out of my depth and though you knew what was best.

Don't feel bad I work in Slackware. If it was Slack I would have his back. Yes I know it rhymes and is a pun as well. I don't utilize much private networking at least not this type of thing

Started on Slackware back in .... 93'ish? Couldn't have been much later. Dropped out by 95 I think... Still have my 0.99 3.5 inch floppies though I'm sure they've got rampant bitrot by now ...

Those were the days. No GUI command line interface by default. 8 hours of downloading to get the OS. Man we were so cool back then. Now we have become.... Ehhhhhh mainstream almost.

FrankZ · December 2022

OP seems gone from the conversation. @tarasis where are you? How are things going now ?

AuroraZero · December 2022

@FrankZ said:
OP seems gone from the conversation. @tarasis where are you? How are things going now ?

Probably tried to hard taking a dump and passed out, but seriously let us know what is going on! This tension is killing me.

tetech · December 2022

Did you check the firewall?

arirang · December 2022

I have a same problem in Amsterdam and On OGF I saw the same problem which he said was on their side.

tarasis · December 2022

Apologies everyone for leaving the conversation. It was 2am and I crashed out, so you were kinda right @AuroraZero. Will have a read of all you have said and let you know status.

yes I have tried rebooting after making changes.
No firewall (yet) on Ubuntu, so it should receive pings without issue.
Yes firewall on Arch, but its open for ping from anywhere.

@skorous said:
When you say 'starting a ping on the Ubuntu side' you mean you have the tcpdump running on Ubuntu side and have Ubuntu try to ping Arch, right? So the Ubuntu machine is seeing it's own traffic ( guessing so since it's knows its name )? What happens if you tcpdump the Arch side and try the same?

Nothing appears in the output from the Ubuntu box

but there is data from other sources

```[root@backup ~]# tcpdump -i ens4
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:06:40.156922 ARP, Request who-has 5. tell backup, length 28
14:06:40.182060 ARP, Request who-has one.one.one.one tell backup.tarasis.net, length 28
14:06:41.163887 ARP, Request who-has 5. tell backup, length 28
14:06:41.190542 ARP, Request who-has one.one.one.one tell backup.tarasis.net, length 28
14:06:42.177211 ARP, Request who-has 5. tell backup, length 28
14:06:42.203886 ARP, Request who-has one.one.one.one tell backup.tarasis.net, length 28
14:06:42.244691 ARP, Request who-has wine. tell backup, length 28


However, tcpdump on the Ubuntu side the output is

root@backup2tb:~# tcpdump -i ens4
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:14:29.756928 IP6 backup2tb > ip6-allrouters: ICMP6, router solicitation, length 16
13:17:38.343637 IP6 fe80::fc22:1ff:fec5:7af > ip6-allrouters: ICMP6, router solicitation, length 16
13:18:11.136796 IP6 backup2tb > ip6-allrouters: ICMP6, router solicitation, length 16
13:25:51.605649 IP6 backup2tb > ip6-allrouters: ICMP6, router solicitation, length 16
13:26:18.537889 IP6 fe80::fc22:1ff:fec5:7af > ip6-allrouters: ICMP6, router solicitation, length 16


So I think that is something for me to look deeper at

> Shouldn't it be ```tcpdump -i ens4```? I've never used a number like that...

It can be, the 2 came from when I did tcpdump -D

[root@backup ~]# tcpdump -D
1.ens3 [Up, Running, Connected]
2.ens4 [Up, Running, Connected]
3.any (Pseudo-device that captures on all interfaces) [Up, Running]
4.lo [Up, Running, Loopback]
5.bluetooth-monitor (Bluetooth Linux Monitor) [Wireless]
6.nflog (Linux netfilter log (NFLOG) interface) [none]
7.nfqueue (Linux netfilter queue (NFQUEUE) interface) [none]
8.dbus-system (D-Bus system bus) [none]
9.dbus-session (D-Bus session bus) [none]
```

Seemingly -i understands the 2 to match the output of -D.

Slackware ... gosh that takes me back. I think I tried it and didn't get on and switched to RedHat around 1995 (we had solaris at uni, I wanted something unix esque as home so I could run some code at home in something "similar" to Uni). Then years later Suse, Ubuntu, Gentoo, Arch.

@AuroraZero said:
Edit: told you it was stupid

Try putting in the real Mac addresses

I was using the Mac reported by the ip -o link | grep 00:22 command. What I'm about to try is dropping the MAC address from the Ubuntu (Netplan) config.

@arirang said:
I have a same problem in Amsterdam and On OGF I saw the same problem which he said was on their side.

Interesting, what OS's are you going between?

tarasis · December 2022

Also thanks everyone for chipping in so far, greatly appreciated.

tarasis · December 2022

@skorous said:

@AuroraZero said:
Stupid question did you restart networking on both machines?

Ah, good question.

My best guess is that they each accidentally got their own private network so they don't see each other.

Thats my worry, and given their setup is basically a black box I will need to contact them I suspect. But want to try and rule out everything I can first.

Thinking was the tcpdump's would show no traffic either way and since ARPs are broadcast no matter how buggered the OS setup was it should've seen it. But again, this isn't my area either.

So given what you said, its interesting that the Ubuntu side is seeing absolutely nothing on that interface but Arch does.

Going to eliminate part of the equation now and set up the Ubuntu box as Arch as well.

cyberpunk · December 2022

Yes, there seems to be an issue with private network in NL location (at least for BF sale storage VPSes). I'm not sure if anyone has an open ticket with HostHatch regarding this, so I've made one.

tarasis · December 2022

Yeah I hadn't yet, cheers @cyberpunk. Should I hold off also opening a ticket? Maybe just mention me in yours?

Just finished installing Arch on new box, and configured everything as per original Arch storage server and still no dice. The new storage server doesn't see anything at all on ens4

[root@backup2TB ~]# tcpdump -i ens4
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes

Where the old box starts seeing things within a few seconds.

cyberpunk · December 2022

@tarasis said:
Yeah I hadn't yet, cheers @cyberpunk. Should I hold off also opening a ticket? Maybe just mention me in yours?

I'm not really sure about internal HH policies and if extra tickets from affected clients actually help or slow down the process instead. Also, there is a good chance that the issue is already known to them, but very low on priority list for fixes, since it doesn't affect much provided service

tarasis · December 2022

@cyberpunk I'll wait then, I'd rather not poke more than I have to. Cheers.

Help with HostHatch private network, can't see the other side

Comments