Fedora rawhide white hat playpen

Not_Oles · May 2020

Yesterday I spun up a rawhide box:

Xeon D-1521
4 cores, 8 threads
32 GB DDR4 ECC
2 x 2 TB SATA
250 Mbps symmetrical, unmetered
Montreal (OVH BHS)
Speed and latency tests (10 Gbps)
Minimal server version; wants more yummy dnf
Ephemeral (a few days, maybe longer, maybe not)
White hat only

What can I learn from you? What can we explore together?

ehab · May 2020

how much?

Not_Oles · May 2020

@ehab said:
how much?

Free. But maybe better to say, "priceless," assuming that learning and community are sufficiently valuable as to be "beyond price."

FlamingSpaceJunk · May 2020

Good luck! Last time I tried to run Rawhide it lasted until I updated, which was about 5 minutes after install.

Not_Oles · May 2020

@FlamingSpaceJunk said:
Good luck! Last time I tried to run Rawhide it lasted until I updated, which was about 5 minutes after install.

Haha! Last time I tried to upgrade a clean Fedora install to Rawhide, systemd's auto reboot failed.

I imagined there might be a Rawhide cowgirl person here who might wanna show me something amazing.

tarasis · May 2020

Free. But maybe better to say, "priceless," assuming that learning and community are sufficiently valuable as to be "beyond price."

I admit curiosity, how does one signup?

Not_Oles · May 2020

@tarasis

Welcome!

Please make an ed_25519 ssh key pair if you have not already done so. Please post the ed_25519 public key in this thread.

It might take awhile, but your post eventually should be acknowledged and your login account magically will begin to function.

If you need help making an ed_25519 ssh key pair, please look here.

If you are concerned about posting your ed_25519 public key in public, please look here. If you remain concerned, please send me the key via PM.

tarasis · May 2020

@Not_Oles said:
@tarasis

Welcome!

Please make an ed_25519 ssh key pair if you have not already done so. Please post the ed_25519 public key in this thread.

Thank you

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF4VyxPSrgB66+a0vizoS8yyNBMOj0v1ln3eVlfb1nbk [email protected]

Not_Oles · May 2020

Just sent you login info.

My ssh public keys are in your .ssh/authorized_keys together with yours. If you don't want that, please feel free to remove my keys.

Not much there yet. Just server base install. But we can add whatever we want.

Not_Oles · May 2020

Anybody else?

Not_Oles · May 2020

Thousands of kiddie login attempts daily. So I spent the morning messing with dnf, fail2ban, and friends. Preliminary testing suggests that I might have got fail2ban working since I now seem to be able to ban myself.

I wonder whether anybody would be interested if I were to write up a post about the fail2ban installation and configuration.

I want to change the server hostname to whitehat. Haha!

Still more fun ahead with further work on sshd, fail2ban, firewalld, nftables. Then container + KVM fun begins. Or webserver. Or whatever.

How about getting two or more users to share a screen session? Or some more contemporary method of getting folks together on the same terminal session?

How come not more people are interested? Would more people be interested if we used Debian sid? What about OpenBSD-current? Something else?

@tarasis What now for you, friend? What would you like?

Not_Oles · May 2020

So I succeeded in changing the name of the server to "whitehat" plus also exploring how systemd includes sshd within the systemwide crypto integration. I successfully disabled password authentication. However, it's not clear to me whether disabling password authentication might somehow break something in or relying on the systemwide crypto integration scheme. Plus there are a couple of environment variables the source of which (i.e., where they are set) I haven't found yet.

Anybody else want in?

Not_Oles · May 2020

A bit more on Rawhide systemwide crypto policy as it impacts sshd_config:

What Rawhide is doing to facilitate systemwide crypto policy seems to include having systemd start sshd with the -D option so sshd doesn't daemonize plus also calling various crypto schemes via the -o option so as to take precedence over whatever crypto schemes might be specified in sshd_config. Also, the config file is split into two files. sshd_config now has an include for a subdirectory. The permissions of the new subdirectory are set such that, without doing extra work, even root cannot cd into that directory.

There is a comment in the new subdirectory config file, 05-redhat.conf, suggesting that the system might no longer pay attention to config file changes, but, when I first looked, before I checked systemd, the comment didn't seem fully clear to me about which parts of the config file still were being followed and which parts were no longer being followed.

What I originally saw also included PasswordAuthentication set to yes both in the original config file and yet again in the new, extra config file in the subdirectory. It kinda looked like somebody really might be trying hard to keep PasswordAuthentication set to yes.

The result of the use of command line options in the way systemd starts sshd plus also the lack of clarity resulting from setting PasswordAuthentication to yes two times in the two different files made me wonder if the system somehow really needed PasswordAuthentication set to yes. Moreover, the system, overall, is in a weird state where configuration files are partly used, and partly overridden, and where the only indication of the scope of the changes seems to be whatever command line options were used by systemd to start the ssh daemon as not-a-daemon.

I can appreciate that systemwide crypto policy might be a very excellent idea. Also, I can appreciate that a huge problem with implementing a systemwide crypto policy is that various upstreams continue to release with configuration files that do not pay attention to downstream's version of what a systemwide crypto policy should look like. Additionally, there is the possibility that OVH might have, during the install, changed a configuration file from the default.

It's important to me that I do not appear to be criticizing Rawhide. I appreciate that they are in a bit of a spot versus various upstreams as they implement what seems like a good idea of having a systemwide crypto policy.

I ended up removing the "PasswordAuthentication yes" line from sshd_config and also setting it to "no" in the new, included configuration file from the subdirectory.

The server seems to be running great for a few days now. So I am guessing I might not have totally busted it. Yet.

Not_Oles · May 2020

I've installed fio, and I've been playing with yabs.sh. I want to understand why yabs.sh seems to set 0 (zero) for the disk speed results when run inside a tempfs RAM disk.

Haha, in a way, yabs.sh reporting 0 is exactly right when reading/writing from/to to memory while measuring what's happening on the spinning rust. But it seems I can't just run yabs.sh in the ramdisk and say, "My ramdisk is even faster than a solid state disk."

On the other hand, bench.sh from bench.monster, when run from a tempfs RAM disk, seems to report the disk speed at the same values it reports for the RAM speed.

The two scripts seem to test differently, so, eventually, I'll figure out the differences. Maybe somebody wants to spill the beans? @Mason @cybertech @vmalware Thanks, guys!

Not_Oles · May 2020

These days it's cool to be a data scientist. So maybe I will do that when I grow up.

A couple of days ago, while googling around for something, I tripped across data36.com's tutorial Data Coding 101 – How to install Python, SQL, R and Bash (for non-devs). This 7 part tutorial is a few years old now, and it's written for Ubuntu 18.04, but its server-oriented approach seemed simple enough for me possibly to understand. I also imagined it might be fun to translate all the tutorial's suggested commands from Ubuntu into Rawhide.

Today, I'm part way through the first of the 7 parts. Python, pip, python-dev, and Jupyter seem to be set up and working. Postgres seems to work too, via psql. Next up are pgAdmin, R and Rstudio. And maybe 6 more pages of tutorial.

mobile · May 2020

Do something with cockpit
Honeyd
Well i guess i has no better idea for rawhide

Not_Oles · May 2020

@mobile said:

Do something with cockpit

Honeyd
Well i guess i has no better idea for rawhide

@mobile Thanks for your ideas! Since Cockpit isn't terminal based I probably never would have thought of it if you hadn't mentioned it. The Cockpit project website says they released new version 219 a few days ago on 13 May.

Honeyd is something I always thought might be fun, but I never tried it. The latest commit seems to be from 2013.

Both Cockpit and Honeyd now are on my list for when I want a day off from Data Science 101.

Do you know about Overthewire?

Thanks again! Greetings from the Sonoran Desert!

vyas · May 2020

@Not_Oles said:

@ehab said:
how much?

Free. But maybe better to say, "priceless," assuming that learning and community are sufficiently valuable as to be "beyond price."

That was a nice answer. Kindness, payable by teaching.

One is tempted to reply with the stock “$1, payable by Fatpal “Kind of statements...

And on that note, best wishes with the project

mobile · May 2020

@Not_Oles said:

@mobile said:

Do something with cockpit

Honeyd
Well i guess i has no better idea for rawhide

@mobile Thanks for your ideas! Since Cockpit isn't terminal based I probably never would have thought of it if you hadn't mentioned it. The Cockpit project website says they released new version 219 a few days ago on 13 May.

Honeyd is something I always thought might be fun, but I never tried it. The latest commit seems to be from 2013.

It's still works so i guess it's something worth to try i guess. most of honeypots project are stalled 3-5 years ago too, from open source side at least.

Do you know about Overthewire?

Yes, i did a lot few years ago for CTF exercise. i stopped doing this after realizing most of my costumer are dumb and it's apparently enough at the point you grasp the concept of opcode/shellcode to exploit most of running systems. heck nowadays you can rip doublepulsar and eternalblue off github for free and still find some workstation easily break with it.
who cares about patching windows xp anyway

Not_Oles · May 2020

@mobile said: most of honeypots

Wow! That's an amazing honey list on Github! So many honey projects! And the second-to-last item on the list, the links to so many honey research papers! I had no idea!

I also had to look up doublepulsar and eternalblue.

Yes, whitehat is a sort of clean, non-blackhat term. But I'm still just a kid. I don't really qualify for any hat. Not even RedHat, since I haven't used it much. I kinda like this RawHide, though. It seems to work great!

Thanks yet again for the additional comment, and especially for the Github link. I'll have to read up a bit on all this stuff when I get a chance.

Not_Oles · May 2020

postgres=> CREATE TABLE test(column1 TEXT, column2 INT);
CREATE TABLE
postgres=> INSERT INTO test VALUES ('Hello', 111);
INSERT 0 1
postgres=> INSERT INTO test VALUES ('World', 222);
INSERT 0 1
postgres=> SELECT * FROM test;
 column1 | column2 
---------+---------
 Hello   |     111
 World   |     222
(2 rows)

postgres=>

Not_Oles · May 2020

Today's Rawhide didn't seem automagically to satisfy the TeX and LaTeX dependencies of the group packaged version of R, so I decided to compile the new R-4.0.0 myself.

If anybody had waited to join me on this playpen server because there weren't compilers yet, well, there are compilers and development libraries now.

Not_Oles · May 2020

My fun today included reading Linux containers in 500 lines of code. This C code needs to be updated for newer kernels, but it looks really fun. The author has written newer stuff in Go.

Not_Oles · May 2020

To implement monitoring via its control panel, it seems that the OVH installer sometimes installs its own custom kernels. There has been some controversy (2016) about this practice, including trademark issues, disclosure issues, and issues surrounding how often the OVH kernels are updated.

Although Fedora seems not presently one of the distributions which run on an OVH kernel (the Control Panel says that monitoring is not available for this server), I am wondering whether OVH might have made changes in the Fedora configuration which prevent dnf from successfully booting updated kernels. Right now the playpen server still is rebooting 5.6.6 even though dnf installed 5.6.13. Last time I checked, I seem to recall similar situations occurring on OVH installed Debian and Ubuntu.

Besides the kernel update situation, I seem to recall something about the OVH Ubuntu 18.04 install having networking configured via systemd instead of netplan, possibly because netplan might not have liked OVH's Layer 2 IPv6 gateway configuration). The OVH Proxmox VE 6 install doesn't have IPv6, maybe for the same reason. IPv6 also didn't seem to work for me on OVH Gentoo or on OVH Slackware.

At a higher level, anybody using a server installed by someone else might want to check to see how the actual install differs, if at all, from the expected install. Awhile back I remember carefully checking an OVH FreeBSD install and finding everything satisfactory.

The situation seems to differ slightly, however, respecting OVH Linux installs. On the good side, I happily can say that my OVH-installed Proxmox server does seem to update its kernel in the expected manner. It's running kernel 5.4.41-1-pve #1 SMP PVE 5.4.41-1 (Fri, 15 May 2020 15:06:08 +0200) on Proxmox version 6.2-4.

Besides checking the OS install, there also are lower level checks which might be a good idea (2019). And then there's the hardware itself. All of which brings me to think, yet again, of Ken Thompson's Reflections on Trusting Trust (1984).

Tonight, I seem to prefer writing this instead of digging into Rawhide to try to discover why the kernels are not updating upon reboot as expected. That's for tomorrow, or for the next day. Or maybe I'll wipe the server. If I put Proxmox here, too, at least the kernel updates might be expected to work because the server specs here are identical to the server that works. Plus I could give away free VPSes again. The server has a lot more power than I need.

It's important that nobody thinks I'm criticizing OVH. They have good reasons for doing things the way they do. My experience since 2014 is that they're right on the ball (an hour or two) whenever anything doesn't work as they expect. They openly and explicitly acknowledge problems, and they give prompt refunds. All this and charging like half as much as anybody else is not bad!

Any votes for Proxmox + more free VPSes on this server? How about OpenBSD? Or NetBSD? Gentoo or Funtoo? Continue with Rawhide? Stop writing these posts since not too many people seem interested? Something else?

Have fun, everybody! Greetings from Mexico!

sonic · May 2020

@Not_Oles said:
Any votes for Proxmox + more free VPSes on this server? How about OpenBSD? Or NetBSD? Gentoo or Funtoo? Continue with Rawhide? Stop writing these posts since not too many people seem interested? Something else?
Have fun, everybody! Greetings from Mexico!

FREE VPS!

Not_Oles · May 2020

Hi @sonic! Thanks for your vote!

More about OVH kernels

I thought, for completeness' sake, to add a couple more links to my mention, above, of OVH's Linux kernels which are customized for OVH monitoring:

Looking at the OVH kernel magic again this morning, it occurred to me that OVH might be considered more Debian-centric and less distribution agnostic than I previously imagined, because OVH seems to provide custom monitoring-enabled Debian kernel images but not other distributions.

Also maybe OVH's kernels now are more up-to-date than some people suggested in the 2016 Hacker News article linked above: there now seem to be approximately weekly updates to the OVH kernels. Today is May 26, 2020. The latest OVH kernel update seems to be 4.19.124/1261958 from this morning 26-May-2020 08:56. -

About what to do next

I still am liking the Proxmox option. Proxmox says that Proxmox 6.2 "is based on Debian Buster 10.4 and a 5.4 longterm Linux kernel and includes updates to the latest versions of the leading open-source virtualization technologies QEMU 5.0, LXC 4.0, Ceph Nautilus (14.2.9), and ZFS 0.8.3."

OVH's Proxmox install seems to use Proxmox' repositories instead of OVH's repositories. If the OVH monitoring-enabled kernel is at 4.19 and Proxmox is at 5.4, Proxmox might be the highest Debian-based kernel version available out-of-the-box via the OVH auto-installer. (I'm not sure which kernel version Ubuntu 20.04 uses, but 20.04 isn't yet available via the OVH installer.)

Stopping posts in this thread.

Not many people seem interested since the thread's topic is limited to OVH and Rawhide and, lately, to kernel versions, instead of . . . duh! . . . Low End VPSes. Although I do not expect to post in this thread any more, I certainly will pay attention to any additional votes or comments.

Thanks, everyone! Have a great day!

Fedora rawhide white hat playpen

Comments

More about OVH kernels

About what to do next

Stopping posts in this thread.