QualityBox 34.x Release Notes

hosting
1

We’ve upgraded QualityBox to run MediaWiki v1.34.x which requires an upgrade to Elasticsearch 6.x from 5.x

This upgrade requires a re-index of all wiki content. Elasticsearch is a major version upgrade to v6.8.12. Old upgrade deploy process is deprecated for manual re-indexing.

Changelog / Features / Versions

  1. GitHub Workflow for Continuous Integration
  2. Add Certbot rule
  3. Add Server Reporting module “GoAccess” including real-time statistics using WebSockets. Use Tokyo Cabinet for on-disk support if you know your distro supports it (override with goaccess-tcb) Default: OFF
  4. add --single-transaction to mysqldump for non-blocking backups
  5. m_ignore_local_mods: default (False) will fail on local mediawiki extension modifcations. Overriding can be useful to add to public.yml for development environments
  6. m_use_subdomains: default (False)
  7. enable_certbot: false If you enable certbot, make sure you add m_httpd_server_admin in your public.yml
  8. enable_goaccess: false
  9. mediawiki_version: “REL1_34”
  10. mediawiki_default_branch: “REL1_34”
    1.php_ius_version: “php72u”
  11. php_debian_version: “7.2”
  12. m_parsoid_version: “master”
  13. SubPageList version: “1.6.1”
  14. Maps version: “7.x”
  15. DisplayTitle version: “{{ mediawiki_default_branch }}”
  16. ExternalData version: “master”
  17. PageForms version: “master”
  18. PageForms config: |
    // If enabled all “red links” will bring up a form chooser
    $wgPageFormsLinkAllRedLinksToForms = false;
  19. InputBox version “master”
  20. ReplaceText version “master”
  21. Collapsible Vector version: “{{ mediawiki_default_branch }}”
  22. SemanticInternalObjects version: “master” legacy_load
  23. SemanticDrilldown version: “master” legacy_load
  24. Admin Links version: “master”
  25. ApprovedRevs version: “master”
  26. ImagesLoaded version: “master”
  27. UploadWizard version: “{{ mediawiki_default_branch }}”
  28. DataTransfer version: “master”
  29. SemanticMeetingMinutes version: tags/1.0.0 legacy_load
  30. Semantic Scribunto version: ~2.1
  31. SemanticCompoundQueries version: ~2.1 Allows for the display of more than one SMW inline query in one results display set.
  32. Semantic Result Formats version: “~3.1”
  33. Semantic MediaWiki version: “~3.1”
  34. Simplify IUS on RedHat
  35. Add pidfile for Debian Apache (compat with apache_buddy)
  36. Add defaults for mod_prefork Apache
  37. post_max_size = 100M
  38. upload_max_filesize = 100M
  39. extension=ldap.so
  40. install Python-apt for Debian
  41. change libselinux-python from state=installed to present
  42. remove duplicate install NTP
  43. correct Subsystem sftp path
  44. fix iteritems to items
  45. default to bool on certain ‘when’ conditions
  46. change python-mysqldb from state=installed to present
    1.elasticsearch version 6.8.12
  47. condense elasticsearch template to essential
  48. change JAVA_HOME to use packaged version in /usr/share/elasticsearch
  49. configure rsyslog for Debian
  50. add ignore_errors: true to HaProxy
  51. add ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 stronger encryption requirements
  52. timeout tunnel 1h timeout to use with WebSocket and CONNECT
  53. IPv6 enabled
  54. remove Disallow: /w/ from .htaccess
  55. create symlinks to /w/ for short-urls
  56. change composer install to ‘update’ to do extensions
  57. Ensure SMW storage setup
  58. Comment Verify metastore index upgraded (deprecate?)
  59. Check for existence of python3
  60. fix server performance shell script path
  61. upgrade from Debian Stretch to Buster in Vagrantfile

Problem Notes:

Cannot allocate memory

I ran into

[meza]# git commit -am 'Add back preventative measure to protect DOS'
-bash: fork: Cannot allocate memory
-bash: wait_for: No record of process 3685
[meza]# pwd
/opt/meza
[meza]# git status
-bash: fork: Cannot allocate memory
-bash: wait_for: No record of process 3685
[meza]# free -m
              total        used        free      shared  buff/cache   available
Mem:          48306        6945       40127          29        1233       40838
Swap:          8191        2748        5443
[meza]# ps --sort -rss -eo rss,pid,command | head

To get out of this condition without rebooting, you can trigger the OOM killer manually as follows:

echo 1 > /proc/sys/kernel/sysrq
echo f > /proc/sysrq-trigger
echo 0 > /proc/sys/kernel/sysrq

Cause / Background

In my case, OS was running out of PID instead of memory, the error message was the same though.

The default value of max PID number is 32768, to view the value, run

cat  /proc/sys/kernel/pid_max

Fix

To change the max pid number, run

echo 100000 > /proc/sys/kernel/pid_max

In my scenario, the root cause was that one java prcocess spawned 18k+ threads( in linux kernel, thread is essentially a process), to find out the thread count of each process, run

ps -eo nlwp,pid,args --sort nlwp

Short URLs

I put in a check for m_mediawiki_prefix_dir (only FamilySearch) to tailor the rewrite rules in .htaccess