We’ve upgraded QualityBox to run MediaWiki v1.34.x which requires an upgrade to Elasticsearch 6.x from 5.x
This upgrade requires a re-index of all wiki content. Elasticsearch is a major version upgrade to v6.8.12. Old upgrade deploy process is deprecated for manual re-indexing.
Changelog / Features / Versions
- GitHub Workflow for Continuous Integration
- Add Certbot rule
- Add Server Reporting module “GoAccess” including real-time statistics using WebSockets. Use Tokyo Cabinet for on-disk support if you know your distro supports it (override with goaccess-tcb) Default: OFF
- add
--single-transaction
to mysqldump for non-blocking backups - m_ignore_local_mods: default (False) will fail on local mediawiki extension modifcations. Overriding can be useful to add to public.yml for development environments
- m_use_subdomains: default (False)
- enable_certbot: false If you enable certbot, make sure you add
m_httpd_server_admin
in your public.yml - enable_goaccess: false
- mediawiki_version: “REL1_34”
- mediawiki_default_branch: “REL1_34”
1.php_ius_version: “php72u” - php_debian_version: “7.2”
- m_parsoid_version: “master”
- SubPageList version: “1.6.1”
- Maps version: “7.x”
- DisplayTitle version: “{{ mediawiki_default_branch }}”
- ExternalData version: “master”
- PageForms version: “master”
- PageForms config: |
// If enabled all “red links” will bring up a form chooser
$wgPageFormsLinkAllRedLinksToForms = false; - InputBox version “master”
- ReplaceText version “master”
- Collapsible Vector version: “{{ mediawiki_default_branch }}”
- SemanticInternalObjects version: “master” legacy_load
- SemanticDrilldown version: “master” legacy_load
- Admin Links version: “master”
- ApprovedRevs version: “master”
- ImagesLoaded version: “master”
- UploadWizard version: “{{ mediawiki_default_branch }}”
- DataTransfer version: “master”
- SemanticMeetingMinutes version: tags/1.0.0 legacy_load
- Semantic Scribunto version: ~2.1
- SemanticCompoundQueries version: ~2.1 Allows for the display of more than one SMW inline query in one results display set.
- Semantic Result Formats version: “~3.1”
- Semantic MediaWiki version: “~3.1”
- Simplify IUS on RedHat
- Add pidfile for Debian Apache (compat with apache_buddy)
- Add defaults for mod_prefork Apache
- post_max_size = 100M
- upload_max_filesize = 100M
- extension=ldap.so
- install Python-apt for Debian
- change libselinux-python from state=installed to present
- remove duplicate install NTP
- correct Subsystem sftp path
- fix iteritems to items
- default to bool on certain ‘when’ conditions
- change python-mysqldb from state=installed to present
1.elasticsearch version 6.8.12 - condense elasticsearch template to essential
- change JAVA_HOME to use packaged version in /usr/share/elasticsearch
- configure rsyslog for Debian
- add
ignore_errors: true
to HaProxy - add ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 stronger encryption requirements
-
timeout tunnel 1h
timeout to use with WebSocket and CONNECT - IPv6 enabled
- remove Disallow: /w/ from .htaccess
- create symlinks to /w/ for short-urls
- change composer install to ‘update’ to do extensions
- Ensure SMW storage setup
- Comment Verify metastore index upgraded (deprecate?)
- Check for existence of python3
- fix server performance shell script path
- upgrade from Debian Stretch to Buster in Vagrantfile
Problem Notes:
Cannot allocate memory
I ran into
[meza]# git commit -am 'Add back preventative measure to protect DOS'
-bash: fork: Cannot allocate memory
-bash: wait_for: No record of process 3685
[meza]# pwd
/opt/meza
[meza]# git status
-bash: fork: Cannot allocate memory
-bash: wait_for: No record of process 3685
[meza]# free -m
total used free shared buff/cache available
Mem: 48306 6945 40127 29 1233 40838
Swap: 8191 2748 5443
[meza]# ps --sort -rss -eo rss,pid,command | head
To get out of this condition without rebooting, you can trigger the OOM killer manually as follows:
echo 1 > /proc/sys/kernel/sysrq
echo f > /proc/sysrq-trigger
echo 0 > /proc/sys/kernel/sysrq
Cause / Background
In my case, OS was running out of PID instead of memory, the error message was the same though.
The default value of max PID number is 32768, to view the value, run
cat /proc/sys/kernel/pid_max
Fix
To change the max pid number, run
echo 100000 > /proc/sys/kernel/pid_max
In my scenario, the root cause was that one java prcocess spawned 18k+ threads( in linux kernel, thread is essentially a process), to find out the thread count of each process, run
ps -eo nlwp,pid,args --sort nlwp
Short URLs
I put in a check for m_mediawiki_prefix_dir
(only FamilySearch) to tailor the rewrite rules in .htaccess