[Fixed?] Tiny, But Heroic Oracle Free Tier VPS Runs Out Of Memory During DNF Upgrade!

Not_Oles · May 2023

Oracle Free Tier Powers The MetalVPS Website

The MetalVPS website is served by a tiny, but heroic Oracle Cloud Free Tier VPS The specs are:

VM.Standard.E2.1.Micro.
OCPU count: 1
Memory (GB): 1
Boot Volume Size: 47 GB
Network bandwidth (Gbps): 0.48
Launched: Sun, Jul 17, 2022, 23:25:22 UTC

The instance has been up the best part of a year:

[opc@instance-20220717-1620 ~]$ uptime
 15:27:35 up 301 days, 16:01,  1 user,  load average: 0.06, 0.08, 0.02
[opc@instance-20220717-1620 ~]$

Recent Downtime Alerts From Hetrix Tools

Recently I started receiving alerts from Hetrix Tools about the MetalVPS website being down briefly, just a few minutes at a time, but repeatedly. I confirmed that I couldn't access the website from my Chromebook during the periods when Hetrix said the site was down. Additionally, my ssh connection to the instance stopped working when Hetrix said the site was down. Luckily, I had redundant backups, so all the files were safe.

Checking the logs suggested that periodic DNF updates were causing the tiny, but valiant VPS to run out of memory. When there was no memory, the web server and the ssh server couldn't work. After the OOM killer decided it was time to kill the DNF update, then the web server and the ssh server could serve again.

Just for fun, I repeatedly ran free -h during a manually initiated dnf upgrade to see what happened. Watch the swap go all the way down to zero:

Swap Drops To Zero During DNF Upgrade

chronos@penguin:~/servers/oracle/metalvps$ ssh m
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Fri May  5 01:56:22 2023 from 187.189.238.1
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       189Mi       364Mi       1.0Mi       126Mi       384Mi
Swap:         1.3Gi       463Mi       896Mi
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       244Mi       236Mi       1.0Mi       199Mi       329Mi
Swap:         1.3Gi       460Mi       899Mi
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       258Mi        50Mi       1.0Mi       371Mi       316Mi
Swap:         1.3Gi       460Mi       899Mi
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       397Mi        50Mi       1.0Mi       231Mi       176Mi
Swap:         1.3Gi       460Mi       899Mi
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       583Mi        47Mi       0.0Ki        49Mi        21Mi
Swap:         1.3Gi       546Mi       813Mi
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       576Mi        46Mi       0.0Ki        56Mi        24Mi
Swap:         1.3Gi       719Mi       640Mi
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       552Mi        57Mi       0.0Ki        70Mi        42Mi
Swap:         1.3Gi       870Mi       489Mi
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       563Mi        42Mi       0.0Ki        74Mi        29Mi
Swap:         1.3Gi       1.0Gi       289Mi
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       559Mi        58Mi       0.0Ki        62Mi        39Mi
Swap:         1.3Gi       1.2Gi       123Mi
[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       604Mi        31Mi       0.0Ki        43Mi       3.0Mi
Swap:         1.3Gi       1.3Gi       0.0Ki

After OOM

[opc@instance-20220717-1620 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          680Mi       166Mi       433Mi       0.0Ki        80Mi       422Mi
Swap:         1.3Gi       484Mi       875Mi
[opc@instance-20220717-1620 ~]$

Memory Issues Fixed By Adding Swap

The memory issues seem to have been fixed by adding additional swap. Here's a link to a tutorial which I found helpful: https://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

The Oracle install already had a swap file, but I added a second, larger swap file. Note that these swap files are within the partition on which the OS is located, not on separate partitions as usually is the case for swap.

Back in the old days, I always could set up machines with plenty of swap. Nowadays, on the clouds, often the VMs come without swap. On HN, I read that swap might be less popular these days since swap could present a security issue. Swap could contain private data which needs protection.

But, by adding a swap file, some tiny VPSes can do big jobs! I seem to remember compiling a ton of software out of pkgsrc on a different, but also tiny Oracle VPS running CentOS. Was it all just a matter of adding some swap and then eating dinner and sleeping while pkgsrc reliably finished its enormous workload on the tiny Oracle VPS?

Here's what the terminal output looked like during the addition of extra swap space followed by a successful dnf upgrade.

[opc@instance-20220717-1620 ~]$ swapon -s
Filename                                Type            Size    Used    Priority
/.swapfile                              file            1392636 461880  -2
[opc@instance-20220717-1620 ~]$ ls -lh /.swapfile 
-rw-------. 1 root root 1.4G Jul 17  2022 /.swapfile
[opc@instance-20220717-1620 ~]$ sudo su -
Last login: Sat May 13 04:36:56 GMT 2023 on pts/0
[root@instance-20220717-1620 ~]# cd /
[root@instance-20220717-1620 /]# df -h .
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/ocivolume-root   36G   11G   25G  31% /
[root@instance-20220717-1620 /]# dd if=/dev/zero of=/.swapfile1 bs=1024 count=2093152
2093152+0 records in
2093152+0 records out
2143387648 bytes (2.1 GB, 2.0 GiB) copied, 39.7653 s, 53.9 MB/s
[root@instance-20220717-1620 /]# ls -l .swapfile*
-rw-------. 1 root root 1426063360 Jul 17  2022 .swapfile
-rw-r--r--. 1 root root 2143387648 May 13 21:22 .swapfile1
[root@instance-20220717-1620 /]# chmod 0600 .swapfile1
[root@instance-20220717-1620 /]# ls -l .swapfile*
-rw-------. 1 root root 1426063360 Jul 17  2022 .swapfile
-rw-------. 1 root root 2143387648 May 13 21:22 .swapfile1
[root@instance-20220717-1620 /]# mkswap /.swapfile1
Setting up swapspace version 1, size = 2 GiB (2143383552 bytes)
[ . . . ]
[root@instance-20220717-1620 /]# swapon /.swapfile1
[root@instance-20220717-1620 /]# swapon -s
Filename                                Type            Size    Used    Priority
/.swapfile                              file            1392636 461528  -2
/.swapfile1                             file            2093148 0       -3
[root@instance-20220717-1620 /]# date -u
Sat May 13 21:30:53 UTC 2023
[root@instance-20220717-1620 /]# dnf upgrade
Oracle Linux 8 BaseOS Latest (x86_64)                          4.9 kB/s | 3.6 kB     00:00    
Oracle Linux 8 Application Stream (x86_64)                     149 kB/s | 3.9 kB     00:00    
Oracle Linux 8 Addons (x86_64)                                  57 kB/s | 3.0 kB     00:00    
Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linu  83 kB/s | 3.0 kB     00:00    
Last metadata expiration check: 0:00:01 ago on Sat 13 May 2023 08:17:46 PM GMT.
Dependencies resolved.
===============================================================================================
 Package                Architecture Version                            Repository        Size
===============================================================================================
Installing:
 kernel-uek             x86_64       5.4.17-2136.318.7.2.el8uek         ol8_UEKR6        112 M
 kernel-uek-devel       x86_64       5.4.17-2136.318.7.2.el8uek         ol8_UEKR6         19 M
Removing:
 kernel-uek             x86_64       5.4.17-2136.317.5.5.el8uek         @ol8_UEKR6       136 M
 kernel-uek-devel       x86_64       5.4.17-2136.317.5.5.el8uek         @ol8_UEKR6        75 M

Transaction Summary
===============================================================================================
Install  2 Packages
Remove   2 Packages

Total download size: 131 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): kernel-uek-devel-5.4.17-2136.318.7.2.el8uek.x86_64.rpm   19 MB/s |  19 MB     00:01    
(2/2): kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64.rpm         30 MB/s | 112 MB     00:03    
-----------------------------------------------------------------------------------------------
Total                                                           35 MB/s | 131 MB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                       1/1 
  Installing       : kernel-uek-devel-5.4.17-2136.318.7.2.el8uek.x86_64                    1/4 
  Running scriptlet: kernel-uek-devel-5.4.17-2136.318.7.2.el8uek.x86_64                    1/4 
  Running scriptlet: kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64                          2/4 
  Installing       : kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64                          2/4 
  Running scriptlet: kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64                          2/4 
  Erasing          : kernel-uek-devel-5.4.17-2136.317.5.5.el8uek.x86_64                    3/4 
  Running scriptlet: kernel-uek-5.4.17-2136.317.5.5.el8uek.x86_64                          4/4 
  Erasing          : kernel-uek-5.4.17-2136.317.5.5.el8uek.x86_64                          4/4 
  Running scriptlet: kernel-uek-5.4.17-2136.317.5.5.el8uek.x86_64                          4/4 
  Running scriptlet: kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64                          4/4 
  Running scriptlet: kernel-uek-5.4.17-2136.317.5.5.el8uek.x86_64                          4/4 
  Verifying        : kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64                          1/4 
  Verifying        : kernel-uek-devel-5.4.17-2136.318.7.2.el8uek.x86_64                    2/4 
  Verifying        : kernel-uek-5.4.17-2136.317.5.5.el8uek.x86_64                          3/4 
  Verifying        : kernel-uek-devel-5.4.17-2136.317.5.5.el8uek.x86_64                    4/4 

Installed:
  kernel-uek-5.4.17-2136.318.7.2.el8uek.x86_64                                                 
  kernel-uek-devel-5.4.17-2136.318.7.2.el8uek.x86_64                                           
Removed:
  kernel-uek-5.4.17-2136.317.5.5.el8uek.x86_64                                                 
  kernel-uek-devel-5.4.17-2136.317.5.5.el8uek.x86_64                                           

Complete!
[root@instance-20220717-1620 /]# date
Sat May 13 21:44:31 GMT 2023
[root@instance-20220717-1620 /]# uname -r
5.4.17-2136.308.9.el8uek.x86_64
[root@instance-20220717-1620 /]# cat /etc/os-release 
NAME="Oracle Linux Server"
VERSION="8.7"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.7"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:7:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.7
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.7
[root@instance-20220717-1620 /]#

There have been no more downtime alerts from Hetrix since the extra swap was added.

Additional Steps

At least a couple more things might need to be done.

First, to persist past reboots, the extra swap needs to be added to /etc/fstab and/or synced with whatever systemd might contribute.
Second, the running kernel (5.4.17-2136.308.9.el8uek.x86_64) doesn't seem to be quite the same as the most recently installed kernel (5.4.17-2136.318.7.2.el8uek.x86_64). So maybe a reboot is needed?

The Bigger Picture

I don't understand why this problem happened. Also, I don't understand why this problem happened at the time it happened, rather than sooner or later.

Could it be that the tiny but heroic VM specs are too small to expect dnf to work?
Did I misconfigure something or make some other mistake which caused dnf to run out of space?
Was the current dnf upgrade unusually big, such that it required more space than expected.
Was the problem caused by something else other than dnf?
Was it correct to add a second swap file? Should there be only one? I could have replaced the existing swap file with another, larger swap file, thus keeping only one swap file.
Maybe /etc/fstab still is meaningful in systemd distros?
Maybe the VPS will survive a reboot?
How applicable is all this to other guys running tiny, but heroic Oracle Free Tier VPSes? Are they all fated to run out of memory after a while?
What about other tiny VPSes on providers other than Oracle and running OSes other than Oracle Linux?

Over the next days, maybe I will get to understand more. Thanks again to Oracle for the nifty, heroic free VPS!

Best wishes! I hope everyone gets the server he wants! 🤩

skorous · May 2023

I have one machine which does that. I just use microdnf instead.

Not_Oles · May 2023

@skorous said:
I have one machine which does that. I just use microdnf instead.

https://github.com/rpm-software-management/microdnf

https://unix.stackexchange.com/questions/649598/what-cant-microdnf-do-compared-to-dnf

skorous · May 2023

@Not_Oles said:

@skorous said:
I have one machine which does that. I just use microdnf instead.

https://github.com/rpm-software-management/microdnf

https://unix.stackexchange.com/questions/649598/what-cant-microdnf-do-compared-to-dnf

Yup, it's ... minimal ... but it works in tiny amounts of ram. YUM/DNF aren't designed for small amounts of ram in my experience. It's just not something they're concerned with.

@Not_Oles said:

Oracle Free Tier Powers The MetalVPS Website

Recent Downtime Alerts From Hetrix Tools

Recently I started receiving alerts from Hetrix Tools about the MetalVPS website being down briefly, just a few minutes at a time, but repeatedly. I confirmed that I couldn't access the website from my Chromebook during the periods when Hetrix said the site was down. Additionally, my ssh connection to the instance stopped working when Hetrix said the site was down. Luckily, I had redundant backups, so all the files were safe.

Checking the logs suggested that periodic DNF updates were causing the tiny, but valiant VPS to run out of memory. When there was no memory, the web server and the ssh server couldn't work. After the OOM killer decided it was time to kill the DNF update, then the web server and the ssh server could serve again.

Just for fun, I repeatedly ran free -h during a manually initiated dnf upgrade to see what happened. Watch the swap go all the way down to zero:

Depending on how quickly you want to run a command, take a look at: watch from the procps rpm.

watch -n1 free -h

It will, once per second, clear the screen and re-run the command. Not useful if you're trying to log. Useful if you wanna watch something while doing something else.

But, by adding a swap file, some tiny VPSes can do big jobs! I seem to remember compiling a ton of software out of pkgsrc on a different, but also tiny Oracle VPS running CentOS. Was it all just a matter of adding some swap and then eating dinner and sleeping while pkgsrc reliably finished its enormous workload on the tiny Oracle VPS?

Definitely one option. Another is stopping OS services you may not be using. Swap FirewallD for writing your own iptables. Disable that tuned. If you've got cockpit running disable that ( assuming you're not using it ).

There have been no more downtime alerts from Hetrix since the extra swap was added.

Additional Steps

At least a couple more things might need to be done.

First, to persist past reboots, the extra swap needs to be added to /etc/fstab and/or synced with whatever systemd might contribute.

Adding it to /etc/fstab will be enough. Systemd uses the fstab file to generate the actual systemd units is actually uses to mount stuff iirc.

Second, the running kernel (5.4.17-2136.308.9.el8uek.x86_64) doesn't seem to be quite the same as the most recently installed kernel (5.4.17-2136.318.7.2.el8uek.x86_64). So maybe a reboot is needed?

Yeah, the kernel is the only thing you can't swap dynamically. (*1)

The Bigger Picture

I don't understand why this problem happened. Also, I don't understand why this problem happened at the time it happened, rather than sooner or later.

Kernel updates tend to be large plus if you've added other services this might just be the time all the rest of your daemons have grown enough you ran out of space.

Could it be that the tiny but heroic VM specs are too small to expect dnf to work?

It Depends (tm).

Did I misconfigure something or make some other mistake which caused dnf to run out of space?

Was the current dnf upgrade unusually big, such that it required more space than expected.

Like I said above, kernel updates tend to be large and they spawn a bunch of other processes to rebuild stuff.

Was the problem caused by something else other than dnf?

In the sense that other daemons were eating up enough memory that it ran out of memory, yes.

Was it correct to add a second swap file? Should there be only one? I could have replaced the existing swap file with another, larger swap file, thus keeping only one swap file.

Having more than one is fine. I generally only have one but have used more than one on many occasions like this one.

Maybe /etc/fstab still is meaningful in systemd distros?

It is as I mentioned above.

Maybe the VPS will survive a reboot?

Hard to know. What else have you done to it?

skorous · May 2023

Rats, forgot my footnote.

1 - You can *patch a kernel if you've got Ksplice/Kpatch/whatever Oracle calls it. In certain circumstances heavy wizards can do funky stuff including swap live kernels according to the ancient texts - I've never done it.

somik · May 2023

@Not_Oles said:

@skorous said:
I have one machine which does that. I just use microdnf instead.

https://github.com/rpm-software-management/microdnf

https://unix.stackexchange.com/questions/649598/what-cant-microdnf-do-compared-to-dnf

Swap out the Oracle Linux OS for Ubuntu 20.04 if possible. It runs better on system with small amount of RAMs. Ubuntu 22.04 also has a minimum RAM of 1 GB, but I did manage to run it along with nginx webserver on 512 MB of ram.

skorous · May 2023

@somik said:

@Not_Oles said:

@skorous said:
I have one machine which does that. I just use microdnf instead.

https://github.com/rpm-software-management/microdnf

https://unix.stackexchange.com/questions/649598/what-cant-microdnf-do-compared-to-dnf

Swap out the Oracle Linux OS for Ubuntu 20.04 if possible. It runs better on system with small amount of RAMs.

If swapping OS is an option, go Debian. Those are my two. If it's greater than ~512M then it's Alma/Rocky. If it's 512M or less then it's Debian. Ubuntu's getting too into that whole snaps thing.

Ubuntu 22.04 also has a minimum RAM of 1 GB, but I did manage to run it along with nginx webserver on 512 MB of ram.

I've still got a couple 256M NAT boxes ( and maybe 128M ) running C7. It's painful but doable. ;-)

somik · May 2023

@skorous said:

@somik said:

@Not_Oles said:

@skorous said:
I have one machine which does that. I just use microdnf instead.

https://github.com/rpm-software-management/microdnf

https://unix.stackexchange.com/questions/649598/what-cant-microdnf-do-compared-to-dnf

Swap out the Oracle Linux OS for Ubuntu 20.04 if possible. It runs better on system with small amount of RAMs.

If swapping OS is an option, go Debian. Those are my two. If it's greater than ~512M then it's Alma/Rocky. If it's 512M or less then it's Debian. Ubuntu's getting too into that whole snaps thing.

The server minimal version of ubuntu does not come with any preinstalled snaps that i know of, so it is almost like debian. Honestly I cannot remember why I did not use debian, but I do remember something was different about debian and I could not configure some app/plugin, which is why I switched to ubuntu permanently.

@skorous said:

Ubuntu 22.04 also has a minimum RAM of 1 GB, but I did manage to run it along with nginx webserver on 512 MB of ram.

I've still got a couple 256M NAT boxes ( and maybe 128M ) running C7. It's painful but doable. ;-)

I used to use Centos but stopped using it due to reliability issues when upgrading OS. I had multiple failed updates and server crashes when upgrading Centos 5 to 6 or from 6 to 7. So switched to using ubuntu.

cmeerw · May 2023

@Not_Oles said: total used free shared buff/cache available
Mem: 680Mi 189Mi 364Mi 1.0Mi 126Mi 384Mi
Swap: 1.3Gi 463Mi 896Mi

Why does this only show 680Mi? It's supposed to be a lot closer to 1Gi than that. Do you, by any chance, have a chunk of memory reserved for kdump?

BTW, I am using a couple of 1 GB VPSes with Ubuntu 22.04 and 1 GB is plenty of memory for those.

And any reason why you don't use the Ampere A1 instances (where you can have up to 24 GB of RAM for free)?

Not_Oles · May 2023

@cmeerw Thanks for your always helpful comments!

@cmeerw said:

@Not_Oles said: total used free shared buff/cache available
Mem: 680Mi 189Mi 364Mi 1.0Mi 126Mi 384Mi
Swap: 1.3Gi 463Mi 896Mi

Why does this only show 680Mi? It's supposed to be a lot closer to 1Gi than that. Do you, by any chance, have a chunk of memory reserved for kdump?

I didn't do anything (of which I am consciously aware) that would reserve a bunch of memory.

BTW, I am using a couple of 1 GB VPSes with Ubuntu 22.04 and 1 GB is plenty of memory for those.

Nothing against Ubuntu. I just used Oracle Linux because I was on Oracle's platform, and it seemed a good idea to enjoy a little exercise with the non-Debian dialect.

And any reason why you don't use the Ampere A1 instances (where you can have up to 24 GB of RAM for free)?

I do have a couple of Ampere instances. They are great! One isn't running anything special but has the Oracle Linux developer flavor so I can play with whatever. The other is running Ubuntu with Bootstrap for an article I wrote last year, Updating A Free Udemy Bootstrap Course On Oracle Cloud Free Tier

So I gotta look into why total memory is only 680 MB. Thanks again for the tip! Much appreciated!

eliphas · May 2023

Big chunk of memory missing. Look for something strange on /var/log/dmesg. For comparison, this is my 1G ubuntu 22.04 free tier:

    eliphas@vps3:~$ grep Memory: /var/log/dmesg
    [    0.068026] kernel: Memory: 880964K/1042956K available (16393K kernel code, 4378K rwdata, 10844K rodata, 3240K init, 6556K bss, 161732K reserved, 0K cma-reserved)
    eliphas@vps3:~$ free -h
                   total        used        free      shared  buff/cache   available
    Mem:           964Mi       277Mi       327Mi       0.0Ki       359Mi       515Mi
    Swap:          482Mi        99Mi       382Mi

This machine is only running NFS server -- using the free disk space of the two AMD instances on the Ampere server

Cheers,

Eliphas

Not_Oles · May 2023

TL;DR It survived reboot! No big increase in total memory. Dunno why not.

Specs

Shape configuration
Shape: VM.Standard.E2.1.Micro
OCPU count: 1
Network bandwidth (Gbps): 0.48
Memory (GB): 1
Local disk: Block storage only

Before reboot

[opc@instance-20220717-1620 log]$ sudo less messages # Quick skim, nothing seemed special
[opc@instance-20220717-1620 log]$ free -h 
              total        used        free      shared  buff/cache   available
Mem:          680Mi       230Mi       161Mi        17Mi       288Mi       324Mi
Swap:         3.3Gi       409Mi       2.9Gi
[opc@instance-20220717-1620 log]$

After reboot

[opc@instance-20220717-1620 html]$ date -u; sudo systemctl reboot
Thu May 18 18:31:13 UTC 2023
[opc@instance-20220717-1620 html]$ Connection to metalvps.com closed by remote host.
Connection to metalvps.com closed.
chronos@penguin:~/servers/oracle/metalvps$ date -u; ping -c 2 metalvps.com
Thu 18 May 2023 06:32:10 PM UTC
PING metalvps.com (129.146.184.250) 56(84) bytes of data.

--- metalvps.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1026ms

chronos@penguin:~/servers/oracle/metalvps$ date -u; ping -c 2 metalvps.com
Thu 18 May 2023 06:33:40 PM UTC
PING metalvps.com (129.146.184.250) 56(84) bytes of data.
64 bytes from 129.146.184.250 (129.146.184.250): icmp_seq=1 ttl=53 time=43.4 ms
64 bytes from 129.146.184.250 (129.146.184.250): icmp_seq=2 ttl=53 time=56.5 ms

--- metalvps.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 43.422/49.965/56.508/6.543 ms
chronos@penguin:~/servers/oracle/metalvps$ date -u; curl https://metalvps.com
Thu 18 May 2023 06:35:14 PM UTC
[ . . . ] # Curl worked. Please grab the site off the web if you want to see it. 
chronos@penguin:~/servers/oracle/metalvps$ date -u; ssh m
Thu 18 May 2023 06:36:50 PM UTC
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Thu May 18 18:21:41 2023 from 187.189.238.1
[opc@instance-20220717-1620 ~]$ date -u; free -h
Thu May 18 18:37:06 UTC 2023
              total        used        free      shared  buff/cache   available
Mem:          680Mi       482Mi        41Mi       1.0Mi       157Mi        94Mi
Swap:         1.3Gi        58Mi       1.3Gi
[opc@instance-20220717-1620 ~]$ sudo swapon /.swapfile1
[opc@instance-20220717-1620 ~]$ swapon -s
Filename                                Type            Size    Used    Priority
/.swapfile                              file            1392636 62720   -2
/.swapfile1                             file            2093148 0       -3
[opc@instance-20220717-1620 ~]$

Post-reboot OS Release

chronos@penguin:~/servers/oracle/metalvps$ ssh m
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Thu May 18 21:38:27 2023 from 187.189.238.1
[opc@instance-20220717-1620 ~]$ uname -r
5.4.17-2136.318.7.2.el8uek.x86_64
[opc@instance-20220717-1620 ~]$ uname -a
Linux instance-20220717-1620 5.4.17-2136.318.7.2.el8uek.x86_64 #2 SMP Fri May 5 20:08:47 PDT 2023 x86_64 x86_64 x86_64 GNU/Linux
[opc@instance-20220717-1620 ~]$ ls /etc/*release*
/etc/oracle-release  /etc/redhat-release  /etc/system-release-cpe
/etc/os-release      /etc/system-release
[opc@instance-20220717-1620 ~]$ cat /etc/oracle-release 
Oracle Linux Server release 8.7
[opc@instance-20220717-1620 ~]$ date -u
Thu May 18 22:26:31 UTC 2023
[opc@instance-20220717-1620 ~]$ logout
Connection to metalvps.com closed.
chronos@penguin:~/servers/oracle/metalvps$

Thanks again to Oracle for the nice VPS on their nice cloud platform!

mwt · May 2023

@somik said:
The server minimal version of ubuntu does not come with any preinstalled snaps that i know of, so it is almost like debian. Honestly I cannot remember why I did not use debian, but I do remember something was different about debian and I could not configure some app/plugin, which is why I switched to ubuntu permanently.

True, though the Ubuntu images on Oracle cloud, in particular, come with a oracle-cloud-agent snap pre-installed.

somik · May 2023

@mwt said:

@somik said:
The server minimal version of ubuntu does not come with any preinstalled snaps that i know of, so it is almost like debian. Honestly I cannot remember why I did not use debian, but I do remember something was different about debian and I could not configure some app/plugin, which is why I switched to ubuntu permanently.

True, though the Ubuntu images on Oracle cloud, in particular, come with a oracle-cloud-agent snap pre-installed.

Oh, right, all OS comes with oracle-cloud-agent as it makes it easier to manage the OS from their OCI console.

mwt · May 2023

@somik said:

@mwt said:

@somik said:
The server minimal version of ubuntu does not come with any preinstalled snaps that i know of, so it is almost like debian. Honestly I cannot remember why I did not use debian, but I do remember something was different about debian and I could not configure some app/plugin, which is why I switched to ubuntu permanently.

True, though the Ubuntu images on Oracle cloud, in particular, come with a oracle-cloud-agent snap pre-installed.

Oh, right, all OS comes with oracle-cloud-agent as it makes it easier to manage the OS from their OCI console.

I wasn't sure if it was a snap on el variants too, but it might be.

somik · May 2023

@mwt said:

@somik said:

@mwt said:

@somik said:
The server minimal version of ubuntu does not come with any preinstalled snaps that i know of, so it is almost like debian. Honestly I cannot remember why I did not use debian, but I do remember something was different about debian and I could not configure some app/plugin, which is why I switched to ubuntu permanently.

True, though the Ubuntu images on Oracle cloud, in particular, come with a oracle-cloud-agent snap pre-installed.

Oh, right, all OS comes with oracle-cloud-agent as it makes it easier to manage the OS from their OCI console.

I wasn't sure if it was a snap on el variants too, but it might be.

I don't think so, but I cant recall exactly either...

Not_Oles · May 2023

People were mentioning that Oracle Cloud VMs running Ubuntu seemed to show more total memory than appeared to be present in my awesome, tiny VM.Standard.E2.1.Micro instance which is serving metalvps.com..

I only have the single VM.Standard.E2.1.Micro instance, but I do also have two identically configured, larger VM.Standard.A1.Flex instances, one of which happens to be running Oracle Linux and the other happens to be running Ubuntu.

So here is a comparison of the total memory reported by free in two identically configured 12 GB RAM Oracle Arm instances, one running Oracle Linux and the other running Ubuntu. As you can see, the results here do seem to show that Ubuntu's free reports more total memory than Oracle's free in identically configured instances.

Any ideas what causes the possible difference?

Thanks to Oracle for the great, free VMs!

Instance-20220614-1523 -- Oracle Linux

From Oracle Cloud Web GUI
Shape configuration
Shape: VM.Standard.A1.Flex
OCPU count: 2
Network bandwidth (Gbps): 2
Memory (GB): 12
Local disk: Block storage only

chronos@penguin:~/servers/oracle/aarm$ ssh -i ~/.ssh/id_ed25519 [email protected]
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Thu May 18 22:34:52 2023 from 187.189.238.1
[opc@instance-20220614-1523 ~]$ free
              total        used        free      shared  buff/cache   available
Mem:       11383936     1665536     6266048      592064     3452352     6811264
Swap:       4194176       33920     4160256
[opc@instance-20220614-1523 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:           10Gi       1.6Gi       6.0Gi       578Mi       3.3Gi       6.5Gi
Swap:         4.0Gi        33Mi       4.0Gi
[opc@instance-20220614-1523 ~]$ cat /etc/oracle-release 
Oracle Linux Server release 8.7
[opc@instance-20220614-1523 ~]$

Instance-20220616-1631 -- Ubuntu

From Oracle Cloud Web GUI
Shape configuration
Shape: VM.Standard.A1.Flex
OCPU count: 2
Network bandwidth (Gbps): 2
Memory (GB): 12
Local disk: Block storage only

ubuntu@instance-20220616-1631:~$ free
               total        used        free      shared  buff/cache   available
Mem:        12226920      632860    10233948        7208     1360112    11393028
Swap:              0           0           0
ubuntu@instance-20220616-1631:~$ free -h
               total        used        free      shared  buff/cache   available
Mem:            11Gi       617Mi       9.8Gi       7.0Mi       1.3Gi        10Gi
Swap:             0B          0B          0B
ubuntu@instance-20220616-1631:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
ubuntu@instance-20220616-1631:~$

cmeerw · May 2023

@Not_Oles said: Any ideas what causes the possible difference?

My bet would still be on kdump being enabled by Oracle Linux. Do you see something like "crashkernel" in the Linux command line (/proc/cmdline)?

Not_Oles · May 2023

@cmeerw Looks like you might be right! Congrats plus double bandwidth!

[opc@instance-20220614-1523 ~]$ date -u
Mon May 22 13:59:34 UTC 2023
[opc@instance-20220614-1523 ~]$ ls -l /proc/cmdline 
-r--r--r--. 1 root root 0 May 22 00:00 /proc/cmdline
[opc@instance-20220614-1523 ~]$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.4.17-2136.318.7.1.el8uek.aarch64 root=/dev/mapper/ocivolume-root ro crashkernel=auto LANG=en_US.UTF-8 console=ttyAMA0 console=ttyAMA0,115200 rd.luks=0 rd.md=0 rd.dm=0 rd.lvm.vg=ocivolume rd.lvm.lv=ocivolume/root rd.net.timeout.carrier=5 netroot=iscsi:169.254.0.2:::1:iqn.2015-02.oracle.boot:uefi rd.iscsi.param=node.session.timeo.replacement_timeout=6000 net.ifnames=1 nvme_core.shutdown_timeout=10 ipmi_si.tryacpi=0 ipmi_si.trydmi=0 libiscsi.debug_libiscsi_eh=1 loglevel=4 ip=single-dhcp crash_kexec_post_notifiers
[opc@instance-20220614-1523 ~]$

ubuntu@instance-20220616-1631:~$ date -u
Mon May 22 14:02:30 UTC 2023
ubuntu@instance-20220616-1631:~$ ls -l /proc/cmdline 
-r--r--r-- 1 root root 0 May 18 22:56 /proc/cmdline
ubuntu@instance-20220616-1631:~$ cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-5.15.0-1035-oracle root=UUID=55055f8a-4546-4ae6-851b-a98cce12b472 ro console=tty1 console=ttyS0 nvme.shutdown_timeout=10 libiscsi.debug_libiscsi_eh=1 crash_kexec_post_notifiers
ubuntu@instance-20220616-1631:~$

[Fixed?] Tiny, But Heroic Oracle Free Tier VPS Runs Out Of Memory During DNF Upgrade!

Oracle Free Tier Powers The MetalVPS Website

Recent Downtime Alerts From Hetrix Tools

Swap Drops To Zero During DNF Upgrade

After OOM

Memory Issues Fixed By Adding Swap

Additional Steps

The Bigger Picture

Comments

Oracle Free Tier Powers The MetalVPS Website

Recent Downtime Alerts From Hetrix Tools

Additional Steps

The Bigger Picture

Cheers,

Specs

Before reboot

After reboot

Post-reboot OS Release

Instance-20220614-1523 -- Oracle Linux

Instance-20220616-1631 -- Ubuntu