Yugabyte Major Upgrade, p4: Actual Upgrade.
TL;DR: Run the check and then upgrade the database in careful steps. All implemented by just replacing containers and running a few (scripted) commands.
Success!
(but I did pop some questions at the bottom)
The upgrade was Plug and Play with containes, more or les...
Background.
After preparation (link), my cluster is ready for upgrade. My cluster looks like this:
Notice three containers running the yb-master-processes, node2, node3 and node4. And another three containers running the yb-tserver processes, node5, node6 and node7 (see prep-steps in earlier blog p3).
I can now follow the required sequence:
a) upgrade masters, by replacing three containers.
b) run the check and the catalog upgrade, using 1 of the new containers.
c) upgrade tservers by replacing three containers.
d) finalize the upgrade.
And if so desired, I can also re-distribute the load over all six containers again.
Replacing the first container, fast.
I will use node2 to introduce the new software-version (docker pull yugabytedb/yugabyte:latest), using these commands:
docker rm -f node2
docker run -d --network yb_net --hostname node2 --name node2 \
-p5432:5433 -p7002:7000 -p9002:9000 \
-p12002:12000 -p13002:13000 -p13432:13433 -p15432:15433 \
-v /Users/pdvbv/yb_data/node2:/root/var \
-v /Users/pdvbv/yb_data/sa:/var/log/sa \
yugabytedb/yugabyte:latest \
tail -f /dev/null
Notice that until I start the yb-master, I am running with only two master-processes, hence higher risk. See suggestion S5 below.
Also note (out of scope): I normally add some scripts and some lines to .bash_profile. Out of scope for this blog, except for that I set the $MASTERS for use with yb-admin and I fetch the flagfiles to use in startup.
First I want to run the check, the command is:
./postgres/bin/pg_upgrade --check -U yugabyte \
--old-host node5 --old-port 5433 \
--old-datadir /root/var/data/pg_data_11
Output in my case:
Looks OK to me.
Now I can proceed to start the yb-master by using the previously created script:
As we can see I am back to a cluster with 3 nodes providing master-services. Fully RF=3. Safe.
Being paranoid, I would also check that my database has been processing SQL-calls without interruption, and would I probably check some more webpages and logfiles.
Replace other containers running yb-master:
When all looks OK, I proceed, and replace node3, and start the yb-master on that node.
Followed by the same process for node4.
Make sure to do these nodes 1 by 1; to ensure that at no point do I loose the quorum of 2 masters.
When that is all done (count a few minutes, but not much more), I have a cluster running in mixed-mode: the masters are on the new version, the tservers are still on the old version.
I am ready for the catalog-upgrade.
Upgrade the Catalog.
The most critical upgrade step for an RDBMS is often the catalog-upgrade. In yugabyte, this is done as follows:
Be patient, this can take time. In my case it took 6 minutes. (see suggestion S6). But the RDBMS kept processing queries normally. My (light!) testload didnt see any noticable hiccup or even slowdown.
Notice that my upgrade command is done from a bash shell inside a (new-version-)container. Could I possibly also do this "from outside the container" (a question for yb? - see Q2 below).
(more comments here? how to monitor this process? see S6 below, and how about the contents of the logfiles ? )
When the upgrade of the catalog is successful, we proceed...
Next: upgrade (replace) containers for the yb-tserver processes.
Upgrading the yb-tservers is done by replacing each container with the new version and starting the processes via yugabyted:
docker rm -f node5
docker run -d --network yb_net --hostname node5 --name node5 \
-p5435:5433 -p7005:7000 -p9005:9000 \
-p12005:12000 -p13005:13000 -p13435:13433 -p15435:15433 \
-v /Users/pdvbv/yb_data/node5:/root/var \
-v /Users/pdvbv/yb_data/sa:/var/log/sa \
yugabytedb/yugabyte:latest \
tail -f /dev/null
# reminder: copy in flagfiles, bashrc etc..
docker exec -it node5 yugabyted start
There will be (short) intervals where some tablets are under-replicated, hence I recommend scripting of the commands involved.
In my cluster, I can start everything using "yugabyted". The original configuration was stored by by yugabyted in /var/root/conf/yugabyted.conf. This conf-file included the location of flagfiles, Hence all my intended customizations and flags for version_compatibility got loaded without problems (I checked...):
Notice how the version is now 25.x (the pg15 version), and the parameter for compatibility is still set to 11 to allow old nodes to participate.
The paranoid will now check: masters, servers, postgres-connections, and test-script still running... Yes, All Good (in my case...)
I then repeat this process for all the remaining nodes, node6 and node7, until no old-version of software is in use in the cluster anymore.
NB: Notice how yugabyted also starts the (not active) master-processes, this is a quirk of yugabyted.
Finalize the upgrade.
To complete the upgrade I issue the command:
This command runs in a few seconds (in my case), and my output looks re-assuring. My ugrade seems to be a Aucces.
There are a few more things to do:
First, I should not forget to remove the compatibility parametr from the flagfiles. To avoid re-starting the components, I can also use a CLI utility to "inject" the value of this parameter into the running processes.
I should run analyze; (as per the doc).
And I can remove the node2, node3 and node4 from the blacklist to have the yb-tservers re-distribute their load over my complete cluster.
Alternatively, I could remove 3 out of the 6 nodes and have my database run on a smaller 3-node clusters. There are Many possibilities.
The Recap.
And that completes the "major version upgrade" of my YBDB cluster. I have achieved the following:
- Zero Downtime for the database.
- Use of only "container images", I did not install any software myself, thus eliminating any risk of install-errors.
- Resilience: The points in time where I didnt have complete RF=3 were brief, with the longest time window being less then 3 mintues for the replacement of the first (master)container. When I did pg_check and start the master from a pre-created script. Improvements here are still possible, see Suggestions below.
- Rollback: I did not test beyond the "happy flow". But being lazy, I probably wont spend too much time in testing the non-happy-flows. I do appreciate the complexity here, and from answers to some questions I noticed YBDB are puttting in quite some time+effort here.
To implement my upgrade, I had to isolate my master-processes on separated nodes and replace those one by one. It all worked smoothly once I understood the concepts and the correct order of the steps to take.
As usual, I am always, curious how others obtain the same results. Feel free to contact me for comments.
-- -- -- initial End of this blogpost -- -- --
Appendix: Some Suggestions and Questions:
I can never resist going over my notes and come up with...
Suggestions:
S5: Use extra master node(s) to reduce risk.
you could reduce risk further with use of one or more additional master-nodes. You can add a container with new software to do the check and start a master-process on that container. That way you can always have 3 master processes active.
The same tricks could apply to the yb-tserver processes, but that would mean transferring data between tservers and it would take more time + resources.
S6: provide feedback on slow steps.
Notably the upgrade of the catalog seemed to take up some time. But this remark also applies to pg_check and finalize upgrade.
It helps the nervous observer if there is some output (on stdout), to show that a running process is not in any serious trouble.
Questions for YB and YB-users:
Q1: How many run RF=5 to further reduce the risk ?
Using RF=5 would (slightly) reduce risk during upgrades or other maintenance activities. But the running of a larger and slightly more complicated cluster in itself is an additional risk-factor.
Q2: Running pg_check and yb-admin commands from ...where?
For some commands, I have to shell into the container (e.g. docker exec -it node2 bash...). Those are: pg_check, upgrade_ysql_major_catalog_version, and finalize_upgrade. Can those also be run from "outside the container", e.g. by just supplying the --master_addresses= ?.
To take it 1 step furthe, IF 1) yugabyted can start just-masters and just-servers, and 2) if all other commands can be given from the outside over the master_address_list, THEN I can see how a containerized upgrade can work (be orchestrated) fairly simple. First by adding new-version (master)containers into a cluster, and then migrating components to those new containers. Then upgrading the catalog. Followed by brief plug-out and plug-in of containers for yb-tserver running new version. I suspect this could be orchestrated by some k8s operator...?
Comments
Post a Comment