Upgrade R1-2021 => R2-2021 "Juniper"

Based on "platform-complete"
with automatic migration of the database contents
Deployment with Okapi, "Single Server" on BS Ubuntu 20.04.2 LTS

Increase server hardware

Required for an upgrade R1 → R2 :
 8 CPUs
 40 GB RAM
recommended: 350 GB SSD HD

Install Ubuntu Updates

  sudo apt-get update
  sudo apt-get upgrade
  sudo reboot


The following has to restart after a system reboot:
- postgres
systemctl start postgresql 
- docker
systemctl start docker.service
- okapi
systemctl start okapi.service 
- nginx
systemctl start nginx 


Upgrade of Okapi Version / Restart Okapi


You have to do that, otherwise you can't pull the new modules.


Installation and configuration of Okapi.

The Okapi version is in the platform-complete/install.json file, so a current version of this file must be downloaded first:

# Clone the platform-complete repository if you haven't already done so
# git clone https://github.com/folio-org/platform-complete

  Change into that directory:
  cd ~/platform-complete
  git fetch
# new branch R2-2021, new Tag R2-2021-GA
# Check out that branch.
# Save your local changes. This should only pertain to stripes.config.js .
# Discard changes to install.json etc:
  git restore install.json
  git restore okapi-install.json
  git restore stripes-install.json
  git restore package.json

  git stash save
  git checkout master
  git pull
  git checkout R2-2021-GA
  git stash pop
  # Read R2 Okapi Version from install.json: okapi-4.8.2

  Get Okapi as Debian-Paket from repository.folio.org 

Importiert the FOLIO signing key, add the FOLIO apt repository and install Okapi (of this version).
First save okapi.conf :
  sudo su
  cp -p /etc/folio/okapi/okapi.conf /usr/folio/upgrade/

  wget --quiet -O - https://repository.folio.org/packages/debian/folio-apt-archive-key.asc | sudo apt-key add -
OK
  sudo add-apt-repository "deb https://repository.folio.org/packages/ubuntu focal/"
  sudo apt-get update
  sudo apt-get -y --allow-change-held-packages install okapi=4.8.2-1 # R2-2021 Okapi Version


 # play back okapi.conf :
  sudo cp -p ~/upgrade/okapi.conf /etc/folio/okapi/okapi.conf
  Port range of Okapi may need to be enlarged (9131-9350); Edit okapi.conf. Look how many ports are alreday in use (sudo docker ps --all | wc)
  At least 60 ports must still be free !


  sudo apt-mark hold okapi
okapi set on hold.

New start Okapi

  sudo systemctl daemon-reload
  sudo systemctl restart okapi.service


 In /var/log/folio/okapi/okapi.log there should be:
INFO DeploymentManager fast shutdown
# and directly after
INFO InternalModule InternalModule starting okapiversion=4.3.3
...
INFO MainVerticle Deploy completed succesfully


# Get the list of modules for this tenant (merely informative):
curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/diku/modules
...
}, {
  "id" : "okapi-4.8.2"
} ]
# 8x edge, 50x folio_ (Frontend), 60x mod- (Backend) = the R1-Modules + the R2-Okapi


Pull Module descriptors from the central repository


  vim ~/folio-install/runbooks/single-server/scripts/okapi-pull.json
  {
    "urls": [
        "https://folio-registry.dev.folio.org"
      ]
  }
  cd ~
  curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @folio-install/runbooks/single-server/scripts/okapi-pull.json http://localhost:9130/_/proxy/pull/modules

2021-08-25T13:11:06,833 INFO  ProxyContext         619855/proxy REQ 127.0.0.1:45316 supertenant POST /_/proxy/pull/modules  okapi-4.8.2
2021-08-25T13:11:17,625 INFO  PullManager          Remote registry at https://folio-registry.dev.folio.org is version 4.7.1
2021-08-25T13:11:17,625 INFO  PullManager          pull smart
2021-08-25T13:11:31,578 INFO  PullManager          pull: 2457 MDs to insert
2021-08-25T13:11:33,823 INFO  ProxyContext         619855/proxy RES 200 26989705us okapi-4.8.2 /_/proxy/pull/modules


Deploy a compatible FOLIO backend and enable for the tenant.

Post Environment variables to be used by te deployed modules in the Okapi environment:

  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_HOST\",\"value\":\"10.9.2.62\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_PORT\",\"value\":\"5432\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_DATABASE\",\"value\":\"folio\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_USERNAME\",\"value\":\"folio\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_PASSWORD\",\"value\":\"folio123\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"KAFKA_HOST\",\"value\":\"10.9.2.85\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"KAFKA_PORT\",\"value\":\"9092\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"OKAPI_URL\",\"value\":\"http://10.9.2.85:9130\"}" http://localhost:9130/_/env;

# Merely informativ: Look, how many docker containers are running now:
sudo docker ps | grep -v "^CONTAINER" | wc -l
# there are running approx. 60 containers of the R1 release.

First change environment variables of mod_pubsub ändern, because of  Kommunikation with Kafka

Set KAFKA_HOST and OKAPI_URL to the real IP address:
  cd ~/upgrade
  # The R2 version of pubsub is mod-pubsub-2.3.3.
  # The R2 2021 versions of the modules are in the file ~/platform-complete/okapi-install.json
  wget https://folio-registry.dev.folio.org/_/proxy/modules/mod-pubsub-2.3.3 -O pubsub-module-descriptor.json
  vim pubsub-module-descriptor.json
  # Adjust in the launch descriptor:
    }, {
      "name" : "KAFKA_HOST",
      "value" : "10.9.2.85"
    }, {
      "name" : "OKAPI_URL",
      "value" : "http://10.9.2.85:9130"
    }, {
      "name" : "SYSTEM_USER_PASSWORD",
      "value" : "***secret***"
    } ],
  curl -X DELETE -D - -w '\n' http://localhost:9130/_/proxy/modules/mod-pubsub-2.3.3
  curl -i -w '\n' -X POST -H 'Content-type: application/json' -d @pubsub-module-descriptor.json http://localhost:9130/_/proxy/modules

  Do the same (without OKAPI_URL) for these Modules

Look up R2 versions of the modules in the okapi-install.json :
  mod-data-import
  mod-data-export-spring
  mod-data-export-worker
  mod-ebsconet
  mod-inventory
  mod-inventory-storage
  mod-invoice
  mod-quick-marc
  mod-remote-storage
  # mod-search ==> don't install, otherwise elasticsearch has to be installed also
  mod-source-record-manager
  mod-source-record-storage


First upgrade and enable mod-pubsub 

  curl -w '\n' -D - -X POST -H "Content-type: application/json" -d '[ { "id" : "mod-pubsub-2.3.3", "action" : "enable" } ]' http://localhost:9130/_/proxy/tenants/diku/install?simulate=true

  curl -w '\n' -D - -X POST -H "Content-type: application/json" -d '[ { "id" : "mod-pubsub-2.3.3", "action" : "enable" } ]' http://localhost:9130/_/proxy/tenants/diku/install?deploy=true\&preRelease=false\&tenantParameters=loadReference%3Dtrue


Finally deploy all backend modules and enable:

- Don't install elasticsearch (et.al. !)
  cd ~/platform-complete
  Remove mod-search and folio_inventory-es from install.json. Also remove mod-kb-ebsco-java, folio_search, edge-caiasoft, edge-dematic, mod-gobi, mod-ebsconet and mod-codex-ekb, if you don't need them
  Remove mod-search from okapi-install.json. Also remove mod-kb-ebsco-java, mod-gobi, mod-ebsconet, mod-codex-ekb.
  Remove folio_inventory-es from stripes-install.json. Also remove folio_search, edge-caiasoft, edge-dematic.
  Remove @folio/inventory-es from package.json. Remove @folio/search.


# *****************************************************************************************
# Deploy the modules one by one. For each module a docker container will be started.
# *****************************************************************************************
  # **********************************************
  # Deploy everthing that has not yet been deployed
  # **********************************************


For a single module it works like this:


  cat > circulation-deployment-descriptor.json <<END
{
  "srvcId": "mod-circulation-22.0.4",
  "nodeId": "10.9.2.85"
}
END
  curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @circulation-deployment-descriptor.json http://localhost:9130/_/discovery/modules


Go through the list okapi-install.json module by module and deploy !


  => Now the R2 containers have been deployed, but not yet enabled.

     In addittion the R1 containers are still running on the system. I.e. at the moment the system is still in the state "R1-2021", except for Okapi, which is already on R2-2021 (but backward compatible).


Enable the R2 backend modules for the tenant

  # Only enable, don't deploy because the container ist running already. You need to post the complete list "okapi-install.json", otherwise the module dependencies will not be resolved:


  curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/okapi-install.json http://localhost:9130/_/proxy/tenants/diku/install?simulate=true\&preRelease=false
  curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/okapi-install.json http://localhost:9130/_/proxy/tenants/diku/install?deploy=false\&preRelease=false\&tenantParameters=loadReference%3Dtrue
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
vary: origin
Content-Type: application/json
X-Okapi-Trace: POST okapi-4.8.2 /_/proxy/tenants/diku/install.. : 200 3670882us

List the deployed modules (only Backend):
  curl -w '\n' -D - http://localhost:9130/_/discovery/modules | grep srvcId | wc

List the enabled modules (Frontend + Backend)
  curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/diku/modules | grep id | wc

Post the list of frontend modules to enable them for the tenant

Enable them but no deploy; they are no containers:
  curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/stripes-install.json http://localhost:9130/_/proxy/tenants/diku/install?simulate=true\&preRelease=false

  curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/stripes-install.json http://localhost:9130/_/proxy/tenants/diku/install?preRelease=false
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
vary: origin
Content-Type: application/json
X-Okapi-Trace: POST okapi-4.8.2 /_/proxy/tenants/diku/install.. : 200 8014004us


Installation Frontend: Build an R2 version of the FOLIO Stripes platform


Installation of Stripes and nginx in a Docker Container
# Quelle: https://github.com/folio-org/platform-complete

Adjust Docker File and nginx Configuration

 cd ~/platform-complete
  vim docker/Dockerfile
    ARG OKAPI_URL=http://10.9.2.85:9130 # change the IP address of the server
    ARG TENANT_ID=diku # change tenant


  vim docker/nginx.conf
server {
  listen 80;
  server_name folio-hbz2.hbz-nrw.de;
  charset utf-8;
  access_log  /var/log/nginx/host.access.log  combined;

  # front-end requests:
  # Serve index.html for any request not found
  location / {
    # Set path
    root        /usr/share/nginx/html;
    index       index.html index.htm;
    include mime.types;
    types {
      text/plain lock;
    }
    try_files $uri /index.html;
  }

  # back-end requests:
  location /okapi {
    rewrite ^/okapi/(.*) /$1 break;
    proxy_pass http://10.9.2.85:9130/;
  }
}


  vim stripes.config.js
    Adjust Tenant and Okapi-URL :
      okapi: { 'url':'http://folio-hbz2.hbz-nrw.de/okapi', 'tenant':'diku' },
      The URL must be reachable from the outside, thus from a browser. The endpoint "/okapi" will be redirected in nginx.conf to the internal port 9130.
    Remove this line (don't install the search app) :
          '@folio/search' : {},
Adjust the logo and the favicon icon in stripes.config.js as you like.

  Build the docker container

  sudo su
  docker build -f docker/Dockerfile --build-arg OKAPI_URL=http://10.9.2.85:9130 --build-arg TENANT_ID=diku -t stripes .
Sending build context to Docker daemon  1.138GB
Step 1/19 : FROM node:15-alpine as stripes_build
...
Step 19/19 : ENTRYPOINT ["/usr/bin/entrypoint.sh"]
 ---> Running in a47dce4e3b3e
Removing intermediate container a47dce4e3b3e
 ---> 48a532266f21
Successfully built 48a532266f21
Successfully tagged stripes:latest

# This will run approx. 15 minutes.

Start the docker container

Redirect Port 80 from the outer world to Port 80 of the container. For SSL, also port 443 needs to be redirected (which is not part of this documentation):
  nohup docker run -d -p 80:80 stripes
  Stop nginx on the server:

  sudo service nginx stop

 # Log in to the docker container
  docker exec -it <id> sh
  check whether the config file has been copied correctly:
  vi /etc/nginx/conf.d/default.conf
  # follow the log of the web server inside the container:
  tail -f /var/log/nginx/host.access.log


Clean up.

This step is important, because otherwise the server might easily go down in performance. Especially during your next upgrade this will be of critical importance !!

Undeploy unused containers:
  cd ~/upgrade
  # Which containers are enabled ?
  curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/supertenant/modules
  curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/diku/modules

  # Which containers are in FOLIO discovery and are running ?
  curl -w '\n' -XGET http://localhost:9130/_/discovery/modules | jq '.[] | .srvcId + "/" + .instId' > dockerps.sh
  # This should contain the module versions of the old release
  # => Backend-Modules (mod-*) of the old release + backend modules of the new release (This should be approx. 120 modules).

  # Delete unused, i.e. not enabled, containers:
  z.B.: curl -w '\n' -D - -XDELETE http://localhost:9130/_/discovery/modules/mod-orders-12.0.1
  you will receive an "HTTP/1.1 204 No Content" for each post.

# List the modules which are being deployed now:
  curl -w '\n' -D - http://localhost:9130/_/discovery/modules | grep srvcId | wc
  60
  # This should only contain the module versions of the new release

  # Compare with the number of running docker containers:
  docker ps --all | wc
     63     788   15088
  # subtract Kafka, Zookeeper and the header line=> 60 containers with backend modules (The Sums are matching).

DONE

Log in as "diku_admin:admin" (or with your sys admin login) on http://folio-hbz2.hbz-nrw.de (replace with your server name).

Check if the old data are still there ! E.g, look in Catalogue or User Managament, if your inventory is still there and if your users are still there.