See also

1. Branches at Gerrit interface: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/ContentTranslation,branches
2. To manually update extension branch: https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Updating_the_submodule (You'll need clean copy of Mediawiki/core)

Testing

Note image tag version from Gerrit patch to be deploy. For eg: gerrit:502964 has 2019-04-11-112002-production tag.

Run it:

docker run -p 4000:8080 docker-registry.wikimedia.org/wikimedia/mediawiki-services-cxserver:2019-04-11-112002-production --it --entrypoint /bin/bash -c config.dev.yaml

Where, config.dev.yaml is local cxserver config file.

Endpoints can be tested at: http://localhost:4000

For example MinT can be tested using:

docker run -p 8989:8989 wikipedia-mt:2023-06-16-042302-production

Config files

Production config stays in:

helmfile.d/services/SERVICE/values.yaml and,

WMF specific config stays in:

deployment-charts/charts/SERVICE/templates/_config.yaml

When anything under chart directory is updated. Bump chart version in deployment-charts/charts/SERVICE/Chart.yaml

Deployment

1. Clone deployment-charts repository (for first time).
2. Do needful changes in config (update image or other configuration changes as needed).
3. Make a CR (Example: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/623475) and after a successful review, merge it.
4. After merge, log in in a deployment server (eg: deploy1001), there is a cron (1 minute) that will update the /srv/deployment-charts directory with the contents from git.
5. Go to /srv/deployment-charts/helmfile.d/services/cxserver.
6. Execute: helmfile -e ${CLUSTER} diff --context 5 This will show the changes that it will be applied on the cluster.
7. Execute: helmfile -e ${CLUSTER} -i apply This will materialize the previous diff in the cluster and also will log into SAL the change.

Secrets

If secrets like API key or token need update, it need to be done via SRE at Private Puppet repository.

To update or new keys, open Phabricator task with details and subscribe SRE clinic duty person. Example: task T284887

Status

This is done using helmfile:

1. Change directory to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/SERVICE on a deployment server
2. Unless you are mid un-applied changes the current values files should reflect the deployed values
3. You can check for un-applied changes with: helmfile -e ${CLUSTER} diff --context 5
4. You can see the status with helmfile -e ${CLUSTER} status

Logs

• Service logs are available in logstash such as 'cxserver-last-24-hours' and other similar dashboard.
• Logs can be access from deploy1002 if needed:

kube_env cxserver eqiad

kubectl logs cxserver-production-6c4f65bc-z6hcb cxserver-production

cxserver-production-6c4f65bc-z6hcb is pod name.

To see all logs:

kubectl logs -l app=cxserver -c cxserver-production

Rolling back changes

If you need to roll back a change because something went wrong:

1. Revert the git commit to the deployment-charts repository.
2. Merge the revert (with review if needed)
3. Wait one minute for the cron job to pull the change to the deployment server
4. Change directory to /srv/deployment-charts/helmfile.d/services/SERVICE
5. Execute helmfile -e ${CLUSTER} diff --context 5 to see what you'll be changing.
6. Execute helmfile -e ${CLUSTER} -i apply where CLUSTER is one of (staging, eqiad, codfw).

When patch with chart reverted, helmfile will pick highest number of chart present. Reverting such change will require pinning desired chart after revert configuration. eg. https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/803873

Access

% mysql --skip-ssl -h m5-master.eqiad.wmnet -ucxserver -p
Enter password: