Yugabyte Major Upgrade, p1: summary

T;DR: I upgraded a running yugabyte database from pg11 to pg15, using just the container images. At RF=3, I used 6 containers. Maybe others can use / comment / improve on this.

Note: this will be part of a multi-blog series. This page will give the summary, and some of the background. If needed, I'll put more details on the actual steps in the next parts.


Summary up front :

From reading the doc-pages, I distilled the principle steps:

a) - Prepare and check. Also add parameters (gflag) for mixed mode.

b) - Upgrade masters and run masters in new version, keep tservers on old version.

c) - Upgrade catalog. I frankly have no clue how that happens in the background, but I  understand this is needed. Every database has this "critical moment" of "catalog upgrade"

d) - Upgrade all tservers, do this 1 by 1 to keep quorum at all times.

e) - Finalize upgrade.

f) - Cleanup, remove parameters (gflag) for mixed-mode to allow complete usage of new features.

From these principles, I could create my scenario of more detailed step. I notably didnt want to install any software versions myself. I wanted to stick to the use of pre-installed container images from Yugabyte. But I did add some of my own scripts to the images, turning them effectively into little servers.


Summary of Actual steps (steps numbered for convenience):

10 - Have at least 6 containers running a cluster.

20 - Use flag-files on all containers and components to run mixed-mode. Safer.

30 - Use blacklist to isolate 3 containers to run master processes. 

40 - Tweak those 3 containers so that they run master-process only. 

50  - Re-create 1 of the master-containers with the new image. Mount the same volumes. Dont start any processes yet. Note: you now have only 2 masters running.. (= slight risk).

60 - Run pg_upgrade --check on container with new version.

./postgres/bin/pg_upgrade --check -U yugabyte \

--old-host node5 --old-port 5433 \

--old-datadir /root/var/data/pg_data_11

70 - If outcome is positive: continue... Otherwise, re-create the container with the old image, and fix any errors/warnings.

80 - Start yb-master on the upgraded container. Check for errors. Monitor. 

Note: return to running with 3 masters ASAP, reduce risk...

 90 - Upgrade the other 2 master-containers, 1 by 1 so as not to loose quorum. 

100 - Monitor.

110 - Upgrade the catalog, using one of the new master-containers:

yb-admin --master_addresses $MASTERS upgrade_ysql_major_version_catalog

Note: this may take minutes... be patient.

120 - Monitor.

130  - Replace the non-master containers 1 by 1 with new image and mount existing data-volumes.

140 - 160 Monitor (there were old steps, but now just Monitor)

170 - Finalize via yb-admin:

 yb-admin --master_addresses $MASTERS finalize_upgrade

180 - Monitor

190 - Remove nodes from blacklist to re-distribute load. 

Note: If you want to run minimalistic cluster: remove additional nodes and run with just 3 nodes..

This concludes the summary, for the curious, and for those in a hurry. 

Note: This Summary is now simplified from earlier version.

I will use this summary myself also to re-run the steps, and I do not need more details. To explain a more about the motives and the details, I'll do more write-up below, and I'll put code-examples in a few more blogpages (if only so I can find them myself later on).

Next is to highlight a few of my reasonings as background info (ling to part-2), and to describe some of the steps in more detail (link to part-3). 

Comments

Post a Comment

Popular posts from this blog

[DRAFT] Yugabyte and Postgres - Finding information.

Image
 TL;DR: [DRAFT] Could not resist creating an ERD of tables to collect metrics. I have put "Session" central to my information. Considering it is the application-connect that initiates the session and is the main "customer" causing all activity in a database. Regarding Session: a Datamodel... Aim is currently to find heavy processes and heavy queries.  Spotting underlying tablets / tables may/should follow from that.  But other may be more interested to dive directly into the storage-layer  (as thew metrics on port 9000 seem to focus on rocksdb) Central, I think, is the "Session". as identified from from pg_stat_activity and from. the ps -ef list on the host. I then built a number of entities (tables) around the Session with the aim of collecting data. Relevant Data (I hope).  Metrics... I did get lost a bit in the huge amount of text coming from ports 7000 and 9000. They seem to provide a lot of rocks-db and program/process info. But I could not (yet) rel...
Read more

[Draft] Yugabyte ASH notes and suggestions.

Image
[DRAFT, not for wider publication] TL;DR: Request for adding ID fields to yb_active_session_history. First, use the tablet_uuid instead of the first string-part of the uuid. Furthermore, add IDs to identify the descriptive texts (but keep the text for human readable). And finally consider defining the Root-request-ID as a separate entity (possibly another view: yb_root_request).  Elaborate .. So far I have created a table to store yb_active_session_history records. The columns in the table are copies of the columns from the view, with two additions: id and host. The id comes from  sequence, and the host is the $HOSTNAME of the node where the tserver runs, e.g. where the record is collected (in future, I would use tsrv_uuid). My Ash records are "under development", but this is my latest: I also built a stored-function to capture the records, which was made to run on every node (every tserver) with 1 or 3 or 10min intervals (insert where not exist...). The resulting data can be...
Read more

yugabyte : Finding the data, in a truly Distributed Database

Image
 TL;DR: I dig deeper into yugabyte to find/verify how the various components work, and where the data is kept. I find, with a little digging, that the data is Indeed Distributed and Replicated. Nice! Background In a previous post, I did some RTFM and identified various components that make up a running yugabyte "universe" or a cluster. Now it is time to find the actual data, and explore some of the tools.  hidden-agenda, wishlist: I would really like to be able to "query" this data straight from psql or cql, but that would require additions by yugabyte similar to how postgres exposes its catalog. In a distributed  environment, this is more of a challenge than in a "monolith" where everything is nicely in 1 place. Setup and Tools My "cluster" consists of 4 to 6 nodes running in docker. The setup is described in an earlier blogpost, and was a good starting point. But by now I begin to see I might want to use specific nodes for master and for tserve...
Read more