Yugabyte Major Upgrade p2: background and self-imposed limitations.

TL;DR: I managed to upgrade yugabyte from pg11 to pg15, using just the container images. Major Upgrades are Necessary, so I better study this process. Using Containers-only is one of my goals. And Zero-Downtime is very desirable for a Database. 

Spoiler: Yes! it all worked. But feel free to use / comment / improve on this.

In this part, I describe some of my reasoning: Notably why I limited myself to container images, treat my containers as "little servers". And of course I wanted to keep the database "open" during upgrade... Zero Downtime!

Note: this is part-2 of a multi-part blog series, I'll put more details on the actual steps, with command and scripts, in the next parts.


PostgreSQL: Vendor- and cloud-independence for my RDBMS.

Data is always the Core of any IT system. And I do not want my data to be hostage in some vendor-specific RDBMS or vendor-controlled cloud. Hence PostgreSQL.

I also am a fan of Distributed Databases, and I think some of them have the right concept: Use Postgres for SQL-engine, and construct a storage-system behind it that does the sharding and replicating. 

If I would recommend any such system, I would prefer it to be vendor-independent and cloud-independent. Open source would be a big Plus as well.

With YBDB, I found such a system:

a) open source 

b) uses PostgreSQL as front-end, and 

c) does the distributing, sharding and replicating that I want to obtain Resilience at the database-level. 

YBDB has loads of other goodies as well. See other blogs and the YB site itself.

Now, one of the main features of such a RDBMS would be zero-downtime upgrades, even between major version. And YBDB seems to put a lot of effort in this.


Upgrading to the next major PG-version(s).

As PostgreSQL evolves, the postgres-like databases like Yugabyte will have to follow and also upgrade to the next and future major versions. For Yugabyte, the challenge is to absorb the future changes into the YBDB cluster. And preferably to do the upgrades on a "running database", e.g. with Zero Downtime.

So far, the yugabyte database system has been using pg11, which is quite old by now. Me and others have been asking for higher pg-versions for yugabyte for a while, but the yugabyte-team is hesitating with the upgrade process as they want this Upgrade to be Truly Ironclad Safe, as you would expect from a database-company.

In testing the YBDB version using pg15 (v25.x.y and above), I found it quite comfortable to use. This version is currently the YBDB "preview" version, hence not the main production version yet.

I did notice from asking questions, that the YBDB team puts a lot of effort into the upgrade process. And I must admit that my simple upgrades of a running database only scratch the surface. But it is good fun and a nice learning excercise.


Container images: pre-installed,  eliminate my own errors.

When container-images are available, I prefer to use those, and not modify them if possible. I especially try to avoid installing software/versions into containers or on servers myself. Notably to eliminate myself as source of errors.

I limit myself to setting parameters (I have a few hobbyhorses - maybe link..), and I also add a few scripts to the YBDB provided containers because I am of course an old DBA/Sysadmin and I want to customize and script a few things that I find useful.

Now that YB can tackle the (major-)upgrade process in general, they can quickly catch up with Postgres version of 15 and above. Mainstream Postgres is currently pg17, and pg18 is in Beta (writing this June 2025).


Requirements: my (self-chosen) Limitations.

On reading the current- and preview-documentation for major upgrades, it is clear that the upgrade is a multi-step activity. Notably: the yb-master processes must be upgraded, all of them. And then there are catalog-upgrade steps to be done before the yb-tservers can also be upgraded. 

The fact that masters of newer version need to run next to tservers of existing version made upgrading from container-images extra complicated, since the container images only hold one single version. 

My self-imposed restrictions were:

1. Container images only. Container-images come with the software (pre-)installed and avoid me to do any "installation" steps myself, thus avoiding room for errors.

2. No Downtime. Keep the database open for normal (OLTP) use. This would also include to keep capabilities such as data-ingest and CDC available at all time. 

Note that sensible other requirements would be: 

3. Rollback-capability (at low-effort and zero-risk). This is something I have not tested yet, but I know it has the attention of the YBDB team.

4. Resilience against outage of components or networks. Also something I have not tested much, but which is mostly covered by multiplying the components (e.g. use more nodes, more servers, more zones). My "test" consists of a sh-script that does permanent-insertion of data via psql. I spot hiccups (waits, not errors) when master-role is switched.


Let's Go...

(scripts in next blog. ;-)

With those requirements set, I studied the YBDB doc(-pages). (link...). I began to understand the upgrade-process: master, catalog, tserver, and keeping the database running during the process. And I was able to prepare and execute the upgrade of my system for a container-only upgrade.


Comments

Post a Comment

Popular posts from this blog

[DRAFT] Yugabyte and Postgres - Finding information.

Image
 TL;DR: [DRAFT] Could not resist creating an ERD of tables to collect metrics. I have put "Session" central to my information. Considering it is the application-connect that initiates the session and is the main "customer" causing all activity in a database. Regarding Session: a Datamodel... Aim is currently to find heavy processes and heavy queries.  Spotting underlying tablets / tables may/should follow from that.  But other may be more interested to dive directly into the storage-layer  (as thew metrics on port 9000 seem to focus on rocksdb) Central, I think, is the "Session". as identified from from pg_stat_activity and from. the ps -ef list on the host. I then built a number of entities (tables) around the Session with the aim of collecting data. Relevant Data (I hope).  Metrics... I did get lost a bit in the huge amount of text coming from ports 7000 and 9000. They seem to provide a lot of rocks-db and program/process info. But I could not (yet) rel...
Read more

[Draft] Yugabyte ASH notes and suggestions.

Image
[DRAFT, not for wider publication] TL;DR: Request for adding ID fields to yb_active_session_history. First, use the tablet_uuid instead of the first string-part of the uuid. Furthermore, add IDs to identify the descriptive texts (but keep the text for human readable). And finally consider defining the Root-request-ID as a separate entity (possibly another view: yb_root_request).  Elaborate .. So far I have created a table to store yb_active_session_history records. The columns in the table are copies of the columns from the view, with two additions: id and host. The id comes from  sequence, and the host is the $HOSTNAME of the node where the tserver runs, e.g. where the record is collected (in future, I would use tsrv_uuid). My Ash records are "under development", but this is my latest: I also built a stored-function to capture the records, which was made to run on every node (every tserver) with 1 or 3 or 10min intervals (insert where not exist...). The resulting data can be...
Read more

yugabyte : Finding the data, in a truly Distributed Database

Image
 TL;DR: I dig deeper into yugabyte to find/verify how the various components work, and where the data is kept. I find, with a little digging, that the data is Indeed Distributed and Replicated. Nice! Background In a previous post, I did some RTFM and identified various components that make up a running yugabyte "universe" or a cluster. Now it is time to find the actual data, and explore some of the tools.  hidden-agenda, wishlist: I would really like to be able to "query" this data straight from psql or cql, but that would require additions by yugabyte similar to how postgres exposes its catalog. In a distributed  environment, this is more of a challenge than in a "monolith" where everything is nicely in 1 place. Setup and Tools My "cluster" consists of 4 to 6 nodes running in docker. The setup is described in an earlier blogpost, and was a good starting point. But by now I begin to see I might want to use specific nodes for master and for tserve...
Read more