More and more things move to a cloud.
Times when people used traditional servers have passed.
Who did not heard such sentences in previous years? So time for me to move to the cloud as well.
Powered by Arm cloud.
Call me old fashioned but I like to self host my services. At least some of them (web, private git repos, mail).
But a few months ago I got an e-mail from OVH:
As part of our project to make our datacentres hyper-resilient, we are starting the transformation of the RBX1 datacentre. Our aim is to modernise our infrastructure, in particular, the building where we house servers hosting all our cloud services. Despite our utmost efforts, this modernisation cannot be carried out without affecting your services.
We are writing to inform you that, as a result, the following Bare Metal servers will be closing down on 31 December 2024
I checked their prices for newer machines and decided that it is a bit too much. So it was time to move somewhere else.
I looked at options and then realised that someone mentioned ‘Free tier’ on Oracle cloud:
For free. I dusted my old account, checked is it “Pay As Yo Go” one and started playing with it.
First try was “let use a quarter”: one cpu core, 6 GB of memory and 50 GB of storage. Turned out that is quite enough to run several websites with small traffic. After some tweaks here and there machine got some testing and works.
This blog is hosted on Oracle cloud for a few weeks now and so far no one complained.
My previous servers were handling my mail in standard way. Postfix as SMTP, Dovecot for IMAP. With added stuff on top like Amavis, ClamAV, SpamAssasin, OpenDKIM etc. etc. And Roundcube for webmail if someone needs.
But my system administrator never were top of the shelf ones. It worked, I tweaked it from time to time etc. So this time decided to go with some “all-in-one” solution.
Went with mailcow - a bunch of mail services running as containers, handling things and providing me with some WebUI for configuration.
Added accounts, aliases, setup IMAP sync jobs so all users had their mail present from previous server. Handled DNS changes and server went online.
In meantime I checked for services I run:
My installation of Forgejo remembers Gitea times. So I took some time, cleaned configuration to get rid of gitea names from it. Now it is running with all old repositories.
Factorio multiplayer went to trash. There is no Linux/arm64 binary available.
Discord bot was Python. Migrated fine.
Other shit? Went through it, killed some, migrated some to other places.
How well will it serve my needs? Time will show. My first server had 4 GB ram and dual core Atom cpu (2 cores, 4 threads). And rotating rust as storage. The last one was i5-750 (4 cores) with 16 GB of memory and rotating rust.
So current duo of 2 cores with 8 GB each should work for some time.
Again this year, Arm offered to host us for a mini-debconf in Cambridge. Roughly 60 people turned up on 10-13 October to the Arm campus, where they made us really welcome. They even had some Debian-themed treats made to spoil us!
For the first two days, we had a "mini-debcamp" with disparate group of people working on all sorts of things: Arm support, live images, browser stuff, package uploads, etc. And (as is traditional) lots of people doing last-minute work to prepare slides for their talks.
Saturday and Sunday were two days devoted to more traditional conference sessions. Our talks covered a typical range of Debian subjects: a DPL "Bits" talk, an update from the Release Team, live images. We also had some wider topics: handling your own data, what to look for in the upcoming Post-Quantum Crypto world, and even me talking about the ups and downs of Secure Boot. Plus a random set of lightning talks too! :-)
Lots of volunteers from the DebConf video team were on hand too (both on-site and remotely!), so our talks were both streamed live and recorded for posterity - see the links from the individual talk pages in the wiki, or http://meetings-archive.debian.net/pub/debian-meetings/2024/MiniDebConf-Cambridge/ for the full set if you'd like to see more.
Again, the mini-conf went well and feedback from attendees was very positive. Thanks to all our helpers, and of course to our sponsor: Arm for providing the venue and infrastructure for the event, and all the food and drink too!
Photo credits: Andy Simpkins, Mark Brown, Jonathan Wiltshire. Thanks!
It's been a while since I've posted about arm64 hardware. The last machine I spent my own money on was a SolidRun Macchiatobin, about 7 years ago. It's a small (mini-ITX) board with a 4-core arm64 SoC (4 * Cortex-A72) on it, along with things like a DIMM socket for memory, lots of networking, 3 SATA disk interfaces.
The Macchiatobin was a nice machine compared to many earlier systems, but it took quite a bit of effort to get it working to my liking. I replaced the on-board U-Boot firmware binary with an EDK2 build, and that helped. After a few iterations we got a new build including graphical output on a PCIe graphics card. Now it worked much more like a "normal" x86 computer.
I still have that machine running at home, and it's been a reasonably reliable little build machine for arm development and testing. It's starting to show its age, though - the onboard USB ports no longer work, and so it's no longer useful for doing things like installation testing. :-/
So...
I was involved in a conversation in the #debian-arm IRC channel a few weeks ago, and diederik suggested the Radxa Rock 5 ITX. It's another mini-ITX board, this time using a Rockchip RK3588 CPU. Things have moved on - the CPU is now an 8-core big.LITTLE config: 4*Cortex A76 and 4*Cortex A55. The board has NVMe on-board, 4*SATA, built-in Mali graphics from the CPU, soldered-on memory. Just about everything you need on an SBC for a small low-power desktop, a NAS or whatever. And for about half the price I paid for the Macchiatobin. I hit "buy" on one of the listed websites. :-)
A few days ago, the new board landed. I picked the version with 24GB of RAM and bought the matching heatsink and fan. I set it up in an existing case borrowed from another old machine and tried the Radxa "Debian" build. All looked OK, but I clearly wasn't going to stay with that. Onwards to running a native Debian setup!
I installed an EDK2 build from https://github.com/edk2-porting/edk2-rk3588 onto the onboard SPI flash, then rebooted with a Debian 12.7 (Bookworm) arm64 installer image on a USB stick. How much trouble could this be?
I was shocked! It Just Worked (TM)
I'm running a standard Debian arm64 system. The graphical installer ran just fine. I installed onto the NVMe, adding an Xfce desktop for some simple tests. Everything Just Worked. After many years of fighting with a range of different arm machines (from simple SBCs to desktops and servers), this was without doubt the most straightforward setup I've ever done. Wow!
It's possible to go and spend a lot of money on an Ampere machine, and I've seen them work well too. But for a hobbyist user (or even a smaller business), the Rock 5 ITX is a lovely option. Total cost to me for the board with shipping fees, import duty, etc. was just over £240. That's great value, and I can wholeheartedly recommend this board!
The two things that are missing compared to the Macchiatobin? This is soldered-on memory (but hey, 24G is plenty for me!) It also doesn't have a PCIe slot, but it has sufficient onboard network, video and storage interfaces that I think it will cover most people's needs.
Where's the catch? It seems these are very popular right now, so it can be difficult to find these machines in stock online.
FTAOD, I should also point out: I bought this machine entirely with my own money, for my own use for development and testing. I've had no contact with the Radxa or Rockchip folks at all here, I'm just so happy with this machine that I've felt the need to shout about it! :-)
Here's some pictures...
It (was) that time of year again - last weekend we hosted a bunch of nice people at our place in Cambridge for the annual Debian UK OMGWTFBBQ!
Lots of friends, lots of good food and drink. Of course lots of geeky discussions about Debian, networking, random computer languages and... screws? And of course some card games to keep us laughing into each night!
Many thanks to a number of awesome friendly people for again sponsoring the important refreshments for the weekend. It's hungry/thirsty work celebrating like this!
Warning: If you're not into meat, you might want to skip the rest of this...
This year, I turned 50. Wow. Lots of friends and family turned up to help me celebrate, with a BBQ (of course!). I was very grateful for a lovely set of gifts from those awesome people, and I have a number of driving experiences to book in the next year or so. I'm going to have so much fun driving silly cars on and off road!
However, the most surprising gift was something totally different - a full-day course of hands-on pork butchery. I was utterly bemused - I've never considered doing anything like this at all, and I'd certainly never talked to friends about anything like it either. I was shocked, but in a good way!
So, two weekends back Jo and I went over to Empire Farm in Somerset. We stayed nearby so we could be ready on-site early on Sunday morning, and then we joined three other people doing the course. Jo was there to observe, i.e. to watch and take (lots of!) pictures.
I can genuinely say that this was the most fun surprise gift I've ever received! David Coldman, the master butcher working with us, has been in the industry for many years. He was an excellent teacher, showing us everything we needed to know and being very patient with us when we needed it. It was great to hear his philosophy too - he only uses the very best locally-sourced meat and focuses on quality over quantity. He showed us all the different cuts of pork that a butcher will make, and we were encouraged to take everything home - no waste here!
At the beginning of the day, we each started with half a pig. Over the next several hours, we steadily worked our way through a series of cuts with knife and saw, making the remaining pig smaller and smaller as we went.
We finished the day with three sets of meat. First, a stack of vacuum-packed joints, chops and steaks ready for cooking and eating at home. Second: a box of off-cuts that we minced and made into sausages at the end of the day. Finally: a bag of skin and bones. Our friend's dog got some of the bones, and Jo turned a lot of the skin into crackling that we shared with friends at the OMGWTFBBQ the next weekend.
This was an amazing day. Massive thanks to my good friend Chris Walker for suggesting this gift. As I told David on the day: this was the most fun surprise gift I've ever received. Good hands-on teaching in a new craft is an incredible thing to experience, and I can't recommend this course highly enough.
There are discussions in development circles about Arm powered laptops since forever. But most of time they do not mention “normal” users. Like your parents, spouses, kids who are not developers. People who turn computer on (cold boot or from suspend does not matter) and expect them to “just work”.
My teenage daughter is one of them. Her current laptop is one of Thinkpad models, previous one was Thinkpad as well. Fedora Linux as operating system serves her needs just fine. But despite my 20 years of work with Arm architecture I am unable to get Arm based laptop for her.
There are several Arm powered laptops on a market:
And all of them have issues when it comes to using Fedora Linux on them.
Thanks to Asahi Linux team we can run Fedora Linux on M1/M2 based Macbooks. Which means second hand market as Apple does not sell those models any more (unless 8GB of ram is enough for you).
There are many things which are not supported:
So you pay for hardware and have features which you cannot use. I use Macbook Pro 2021 (with M1 Pro cpu) for local development and stopped checking how work goes.
Qualcomm managed to convince Microsoft to not offer licenses for other vendors which means all we can have are Snapdragon based laptops. Which may work nice under MS Windows but if you want to use Linux then “good luck” is all you can get from me.
Some things work, some do not. I was told that Thinkpad x13s is one of best supported models. Johan Hovold has a Thinkpad x13s status page which lists what works and what needs to be done to have some kind of working laptop.
Definitely not a system for daily use for normal Linux user.
Laptop to run web browser and Android apps. If this is all you need then go for it. But avoid if you are “normal” user and want to run Linux.
Finding how to enable running anything other than ChromeOS may involve digging through Internet pages, finding how to override ‘write protection’ etc.
Just no. Also Arm ones are usually ram limited.
Those are systems for developers only. Normal users should avoid using them as those systems require someone who knows how to prepare them to work at all.
Find/build proper firmware, put it properly in device (SPI Flash or storage media), keeping things up-to-date may end with partially not working device etc.
For developers those are ‘issues’ to workaround/solve but for normal users it may be ‘update went in background and now all I have is black screen’.
And like with Chromebooks you may be limited by ram size (Pinebook Pro has only 4GB ram).
If you are a normal user who wants to run Linux on a laptop then maybe stay away from Arm powered ones. Leave them for developers and check once/twice per year to see how situation looks.
At work I spend most of time on SBSA Reference Platform. Especially in firmware part (Arm Trusted Firmware also known as TF-A and Tianocore EDK2 also known as UEFI). However, for some time, I have felt the need to experiment with some UEFI-related task on existing hardware.
I first searched for “affordable” SystemReady SR system. But options were either Ampere Altra or NVIDIA Grace, both prices at 3000 EUR or more.
So I looked at the budget market and bought a FriendlyELEC NanoPC-T6 SBC.
The FriendlyELEC NanoPC-T6 is a SBC (Seriously Bad Computer) based on Rockchip RK3588 SoC. It has some interesting on-board features:
It comes with metal case which works also as a heatsink.
As you know I expect a good “out of the box” experience. And NanoPC-T6 was like any other SBC I used in the past — unpleasant, horrible and frustrating.
The device came with a fork of U-Boot 2017.09, configured in such terrible way that it was incapable of booting any standard distro images I tried. I managed to boot the pre-installed Android 12 on the eMMC but quickly rebooted to avoid dealing with it.
I managed to boot Debian ‘testing’ manually but there was no networking available under 6.9.x kernel.
I Then moved on to other things as my schedule was quite busy.
This week I reserved some time to get NanoPC-T6 running properly. I downloaded a Rockchip tool called “upgrade_tool” and used it to flash a UEFI image from the EDK2-RK3588 project.
Experience was much, much better. The firmware was now capable of booting distro images, allowed me to choose between ACPI or DeviceTree for hardware description and had proper EFI Shell — almost like a well-developed systems.
I went with ACPI mode and booted directly to Fedora ‘rawhide’ system stored on a USB drive. Linux 6.11-rc booted, found devices plugged into USB 3 ports, recognized both network interfaces (Realtek 8125 ones) and the NVME drive as well. There was video output on the HDMI screen (in a hardcoded 1080p resolution).
I then copied the system from the USB3 drive to the NVME, set the proper boot order and enjoyed a nicely working system.
But aren’t Seriously Bad Computers (SBCs) expected to run with DeviceTree? ACPI is for MS Windows, not for Linux or *BSD systems, right?
So I decided to boot into DT land. It took me a while as I had to remind myself how it works and ensure that UEFI firmware will use the 6.11-rc DTB instead of one for 5.10-rk or 6.1-rk vendor kernels.
Finally it booted — or rather, it “kind of” booted…
No USB, no PCIe == no NVME == boot into emergency mode because the root filesystem is not present…
What will future bring? I am going to find out. I have ordered a Wi-Fi card for m.2 type E slot to see how it performs and I am going to spend some time around this EDK2 fork to make some experiments on real hardware.
Nov 26th saw the release of 4.4.165, 4.9.141, 4.14.84 and 4.19.4
For these LTS kernel versions, results were reported upstream, no regressions were found.
2018-11-26: Rafael Tinoco – bug 4043 – Asked Greg to backport a fix for v4.4, Sasha forwarded to the mm list.
For Android Kernels, regressions were detected.
Issues:
No Others Regressions: 4.4.165 and 4.9.141 on Android 9.
X15: 4.14.84 + O-MR1 – Baselining activity has been particularly effective over the past two weeks, dropping the number of errors from 65 failing tests to 16 as of today. That’s really good progress towards setting a clean baseline.
Bug 4033 Sumit has been looking at the failing CtsBluetoothTestCases android.bluetooth.cts.BluetoothLeScanTest#testBasicBleScan and android.bluetooth.cts.BluetoothLeScanTest.testScanFilter failures.
These tests both pass across all kernels with 8.1. They however fail with both 9.0 and AOSP. Looking at historical AOSP results it appears that failures there started approx in the September timeframe.
Last, successful test builds and test boot to UI with 4.4.165 and 4.9.141 with Android 9) using the newly released clang-r346389 compiler.
Nov 26th saw the release of 4.4.165, 4.9.141, 4.14.84 and 4.19.4
For these LTS kernel versions, results were reported upstream, no regressions were found.
2018-11-26: Rafael Tinoco – bug 4043 – Asked Greg to backport a fix for v4.4, Sasha forwarded to the mm list.
For Android Kernels, regressions were detected.
Issues:
No Others Regressions: 4.4.165 and 4.9.141 on Android 9.
X15: 4.14.84 + O-MR1 – Baselining activity has been particularly effective over the past two weeks, dropping the number of errors from 65 failing tests to 16 as of today. That’s really good progress towards setting a clean baseline.
Bug 4033 Sumit has been looking at the failing CtsBluetoothTestCases android.bluetooth.cts.BluetoothLeScanTest#testBasicBleScan and android.bluetooth.cts.BluetoothLeScanTest.testScanFilter failures.
These tests both pass across all kernels with 8.1. They however fail with both 9.0 and AOSP. Looking at historical AOSP results it appears that failures there started approx in the September timeframe.
Last, successful test builds and test boot to UI with 4.4.165 and 4.9.141 with Android 9) using the newly released clang-r346389 compiler.
Nov 26th saw the release of 4.4.165, 4.9.141, 4.14.84 and 4.19.4
For these LTS kernel versions, results were reported upstream, no regressions were found.
2018-11-26: Rafael Tinoco – bug 4043 – Asked Greg to backport a fix for v4.4, Sasha forwarded to the mm list.
For Android Kernels, regressions were detected.
Issues:
No Others Regressions: 4.4.165 and 4.9.141 on Android 9.
X15: 4.14.84 + O-MR1 – Baselining activity has been particularly effective over the past two weeks, dropping the number of errors from 65 failing tests to 16 as of today. That’s really good progress towards setting a clean baseline.
Bug 4033 Sumit has been looking at the failing CtsBluetoothTestCases android.bluetooth.cts.BluetoothLeScanTest#testBasicBleScan and android.bluetooth.cts.BluetoothLeScanTest.testScanFilter failures.
These tests both pass across all kernels with 8.1. They however fail with both 9.0 and AOSP. Looking at historical AOSP results it appears that failures there started approx in the September timeframe.
Last, successful test builds and test boot to UI with 4.4.165 and 4.9.141 with Android 9) using the newly released clang-r346389 compiler.
Nowadays we have only one Connect per year. And this year we met in Madrid, Spain.
How did it go for me? Good, better than the previous one.
I attended most of the talks I wanted to see, spoke with countless people, and met most of the people on my “to meet” list.
For me, the main topic at Linaro Connect was SystemReady. There was a track for SystemReady IR on the first day and the next days included additional presentations.
Jon Humphreys from Texas Instruments gave an introduction what SystemReady. Explained how systems operated before it, the definition of “Just Works”, and some aspects of testing. He then discussed issues with the current certification process, such as missing firmware files used during testing and problems with errata documents. He also offered recommendations on how to improve the situation.
Vincent Stehlé from Arm spoke a lot about of history and timelines of involved specifications, the U-Boot features timeline and several statistics about SystemReady certifications along with lessons learned.
30.9% of certified systems were for IR band (only SR was higher with 36%), mostly for older versions of the specification.
We learned that there are three certification labs besides Arm itself. And the plan is to eventually remove Arm from this role.
It was interesting to see which distributions were used during IR certification. Fedora Linux leads with 33.9%, followed by openSUSE at 33% and Debian at 21.1%. Other distributions include Ubuntu, SLES, Rocky Linux and RHEL.
Ilias Apalodimas from Linaro presented the history of U-Boot getting features required for SystemReady IR support. He highlighted some common issues and offered suggestions for improvement.
This talk nicely expanded on the topics mentioned in the previous two talks.
Ilias Apalodimas, Pere Garcia Gutiérrez and Peter Robinson presented conclusions from SystemReady’s Advisory Committee workshop.
One idea was that SoC vendors should provide BSP setup in a way that the resulting firmware would already be SystemReady compliant. This would make it easier for OEM/ODM vendors, with some following their own paths while others adhered to the guidelines.
Caleb Connolly from Linaro shared the story how Qualcomm support in U-Boot improved from terrible to good. He mentioned several quirks and issues encountered along the way.
Dong Wei from Arm provided updates on changes to the SystemReady specifications.
I somehow missed that session. It was on my list, but I got distracted by a discussion.
He explained how changes are managed and reminded us how SystemReady bands depend on specifications. Then presented planned changes to BSA, such as creation of VBSA for virtual environments and increase of requirements for future versions.
Another change mentioned was that the SystemReady LS and LBBR recipe of BBR might be deprecated due to lack of interest.
There will be some changes for the SBSA requirements. So far there are no plans to create SBSA level 8, but if a silicon vendor can fulfil all future requirements, Arm may consider publishing SBSA level 8.
The BBSR (Boot Base Security Requirements) specification may become a requirement for future SystemReady versions.
I gave a talk presenting what we achieved in the past year regarding the SBSA Reference Platform in QEMU, Trusted Firmware and EDK2 projects.
I discussed changes to virtual hardware, firmware and the methods we use to transfer configuration data between layers. I explained why I use “Datatree” term instead of “Devicetree” and outlined our plans for the future.
I attended other sessions as well and watched several ones I missed due to discussions or schedule conflicts. Here I want to write about some of them.
Leif Lindholm provided an introduction and updates on the TianoCore EDK2 project.
He covered which specifications it implements, how repositories are structured, the licenses used and the communication methods. Also discussed updates to SSL/TLS libraries and toolchain profiles, contribution rules and recent changes to there rules.
Additionally, he outlined plans for changes to edk2-platforms such as creating stable tags and implementing automated CI.
Johan Hovold shared information about the state of Linux on the Lenovo Thinkpad x13s laptop (which comes with Windows for Arm by default).
The good part is that most of the support is already merged into upstream projects and the required kernel configuration changes have been included in several distributions.
The bad part? There is still a lot of work to be done. I wonder when vendors will learn how to write proper ACPI tables (x13s uses Devicetree to run Linux).
Arnd Bergmann from Linaro spoke about his work on rethinking the Linux kernel system call entry.
He covered how system calls are currently defined now in the Linux kernel. Then showed how he changed it to be more readable.
I have my system calls table but had never looked how kernel code for them looks. Turned out that it can be quite intimidating with macros expanded to macros resulting in complex C code.
Arnd showed that it can be made readable. I hope that his work will land in the Linux kernel soon.
Linaro Connect MAD24 was a success. I got ideas for blog posts, met people who read my posts.
And if you want to watch more talks from conference then there is official Linaro Connect MAD24 playlist on YouTube.
Visit Linaro Resources Hub for video recordings and presentation slides.
There are memberships I forgot about. Some of them remind from time to time with “we have changed rules” mail. Which usually are moments when I remove account from their system.
And there are memberships I remember never mind if I use them on not anymore.
One of them is OpenEmbedded.
I started using OpenEmbedded around February/March of 2004. Will not cover history here as I wrote several posts closer to those years:
I learnt a lot, helped people and companies, got paid by both people and companies, mentored new users and developers.
And got some friends in the OpenEmbedded community.
In 2007 we were discussing about creating official organization. During FOSDEM 2008 a group of developers met and it happened.
First it was OpenEmbedded e.V. based on Germany law, years later it became one of Software in the Public Interest (SPI) projects.
Developers joined, became members, attended meetings to vote etc..
During years, due to member being spread all over the globe, we moved from in-person meetings (with potential proxies) to online voting system.
And in last week I got an email with voting invitation. Of course, I knew that it will come — we have openembedded-members mailing list for a reason ;D
For me my OpenEmbedded membership has highest value when it comes to my memberships. Sure, mostly for sentimental values but when I am at FOSDEM, I always visit OpenEmbedded stand, usually see some OE related talk in the Embedded devroom etc.. Similar at other events.
And my e-mail address in the openembedded.org domain still works. I do not use it but it has value to me.