Mini Course Docs: Tutor Fork Quickstart with eduNEXT/tvm#

Latest update: May 23, 2022 6:06 PM (Please change the versions accordingly if this post is too old.)

1. Set up Environment#

Collect the versions for the software you will be using.

For example, my current setup for Maple uses the following.

Tutor: 13.2.2

Python: 3.8.12

edx-platform: https://github.com/overhangio/tutor/blob/v13.2.2/tutor/templates/config/defaults.yml#L51

TVM: 1.0.0

(Choose according to your release)

1.1. Create Working Directory#

Create a folder to store your Tutor environment.

mkdir tutor-maple
cd tutor-maple

The following examples assume you are running the code from the ~/tutor-maple directory. I will also refer to this directory as $(pwd). You should replace $(pwd) with the result of running the following code in your working directory.

pwd

1.2. Install TVM#

You should now install the TVM package.

pip install git+https://github.com/eduNEXT/[email protected]

1.3. Install Tutor#

The TVM enables you to install Tutor with a simple command.

tvm install v13.2.2

1.4. Create your Environment#

You can now use the Environment Manager function of tvm to create your work environment.

tvm project init

1.5. Enable your Environment#

You need to enable the environment after creating it.

source .tvm/bin/activate

1.6. Install Required Plugins#

You need to install the MFE plugin at this point for the installation to succeed.

tvm plugins install mfe

You can install additional Python plugins by running the tvm plugins install command with the plugin name. The list of official plugins are here: https://overhang.io/tutor/plugins

1.7. Save Configuration#

You should run the save configuration command now to generate the Tutor environment at your working directory. We do not want to start Tutor yet, so we will only create the configuration files.

tutor config save --interactive

You should answer first question as no since you will be doing development.

1.7.1. (Optional) Initialize Git Repository#

You can initialize a git repository inside your working directory at this point to store your Tutor configuration. You should be careful about uploading this git repository as it will include all your database passwords and other secretes. This repository should be a ``private`` repo if you have to upload it.

git init

1.7.2. (Optional) .gitignore#

You might want to create a .gitignore file, so that you do not include the Tutor data files.

volumes
data*
.tvm

1.8. Additional Tutor Configuration#

You should add the following values to the config.yml located inside your Tutor environment created in 1.4. Create your Environment . (You can find it by running echo $(pwd) if you lost it.)

LOCAL_PROJECT_NAME: tutor_maple
DEV_PROJECT_NAME: tutor_maple_dev

Your Tutor installation will be designated the name tutor in your Docker environment by default, if you do not change it here, you will overwrite that Docker environment.

Do not forget to save your config.yml file by running tutor config save.

2. Build Development Image#

You are now ready to create the openedx and openedx Development images.

2.1. Open edX Image#

You can now run the familiar quickstart command.

tutor local quickstart

Once you confirm that your site is up and running, stop it.

tutor local stop

You have now downloaded the openedx image and are ready to build the development image.

2.2. Open edX Development Image#

The development image is created by simply running the following code.

tutor dev dc build lms

After the image is build, you can start your development instance in the detached mode.

tutor dev start -d

Once you confirmed that your development site is up and running, stop it.

tutor dev stop

2.3. (Optional) Tag Your Images#

You should now tag your images to upload them to your image repository if you will be using different machines for development and production.

DOCKER_IMAGE_OPENEDX: <IMAGE REPO>/<IMAGE NAME>:<IMAGE TAG>
DOCKER_IMAGE_OPENEDX_DEV: <IMAGE REPO>/<DEV IMAGE NAME>:<IMAGE TAG>

You should now tag your development image.

2.3.1. (Optional) Create Development Image Again to Tag it#

(You can also find the image ID from Docker Desktop and use the Docker CLI to this step.)

tutor dev dc build lms

2.3.2 (Optional) Upload Your Image#

You can run the following commands to upload your images.

tutor images push openedx
docker push $(tutor config printvalue DOCKER_IMAGE_OPENEDX_DEV)

1. Creating Development Environment#

You are now ready to start working on your own edx-platform fork.

3.1. Create Your Fork#

Use GitHub to create a fork of the ``edx-platform``.

https://github.com/openedx/edx-platform

3.2. Download Your Forked Repo to Your Development Environment#

You are now ready to use your forked repo for development.

First receive the files from the Tutor installation.

mkdir -p $(tutor config printroot)/volumes/edx-platform
tutor dev copyfrom lms /openedx/edx-platform $(tutor config printroot)/volumes/

This way you will be able to use the development tools that are installed inside the Development image. It will take some time to copy over the files.

Then change the remote to point to your fork.

cd $(pwd)/volumes/edx-platform
git remote set-url origin <YOUR_REPO_URL>

Sync your repo.

git fetch

Finally change to the branch you will be using.

git checkout -b <USERNAME>-dev

3.3. Point Tutor Towards Your Forked Repo#

You should create an override file docker-compose.override.ymlat this location $(pwd)/env/dev/docker-compose.override.yml with the following content.

You should replace the variables with your working directory.

version: "3.7"
services:
  lms:
    volumes:
      - "<PWD>/volumes/edx-platform:/openedx/edx-platform"
  cms:
    volumes:
      - "<PWD>/volumes/edx-platform:/openedx/edx-platform"
  lms-worker:
    volumes:
      - "<PWD>/volumes/edx-platform:/openedx/edx-platform"
  cms-worker:
    volumes:
      - "<PWD>/volumes/edx-platform:/openedx/edx-platform"

3.4. Start Development#

Your Tutor installation will now use your fork.

tutor dev start

Congratulations, you now have a development environment to start working on an edx-platform fork.

4. Build your Custom Image#

You have done your changes and are now ready to build your custom image.

4.1. Apply the Patches#

You need to make sure you have all the patches applied to your codebase because we will remove them to prevent problems.

You can find the current patches from the Dockerfile at the $(pwd)/env/build/openedx/Dockerfile directory.

The patches section will start with the following code.

# Identify tutor user to cherry-pick commits
RUN git config --global user.email "[email protected]" \\
  && git config --global user.name "Tutor"
# Patch edx-platform

You can directly run the codes after the RUN command in your edx-platform fork root directory (which is $(pwd)/volumes/edx-platform if you use the one suggested in this guide).

For example, Tutor 13.2.2 uses the following patches.
# Fix forum notification for questions
# <https://github.com/openedx/edx-platform/pull/29611>
RUN git fetch --depth=2 <https://github.com/open-craft/edx-platform/> 03731f19459e558f188c06aac5cc9ca1bbc675c2 && git cherry-pick 03731f19459e558f188c06aac5cc9ca1bbc675c2
# SAML security fix
# <https://github.com/overhangio/edx-platform/tree/overhangio/sec-fix-saml-vulnerability>
RUN git fetch --depth=2 <https://github.com/overhangio/edx-platform/> 3b985f207853e88090d68a81acd52866b71f5af7 && git cherry-pick 3b985f207853e88090d68a81acd52866b71f5af7
# Rate limiting security fix
# <https://github.com/overhangio/edx-platform/tree/overhangio/sec-rate-limiting>
RUN git fetch --depth=2 <https://github.com/overhangio/edx-platform/> b5723e416e628cac4fa84392ca13e1b72817674f && git cherry-pick b5723e416e628cac4fa84392ca13e1b72817674f
# Fix studio-frontend by pinning the installed version
# <https://github.com/openedx/edx-platform/pull/30309>
RUN git fetch --depth=2 <https://github.com/uetuluk/edx-platform/> 53ea60eee86e094f35815ac1c4114d6811f4d458 && git cherry-pick 53ea60eee86e094f35815ac1c4114d6811f4d458

You simply need to run the following commands in your ``edx-platform``
root directory.

git fetch --depth=2 <https://github.com/open-craft/edx-platform/> 03731f19459e558f188c06aac5cc9ca1bbc675c2 && git cherry-pick 03731f19459e558f188c06aac5cc9ca1bbc675c2
git fetch --depth=2 <https://github.com/overhangio/edx-platform/> 3b985f207853e88090d68a81acd52866b71f5af7 && git cherry-pick 3b985f207853e88090d68a81acd52866b71f5af7
git fetch --depth=2 <https://github.com/overhangio/edx-platform/> b5723e416e628cac4fa84392ca13e1b72817674f && git cherry-pick b5723e416e628cac4fa84392ca13e1b72817674f
git fetch --depth=2 <https://github.com/uetuluk/edx-platform/> 53ea60eee86e094f35815ac1c4114d6811f4d458 && git cherry-pick 53ea60eee86e094f35815ac1c4114d6811f4d458

4.2. Remove the Patches#

Now that you applied the patches, you do not need them anymore, use the following v1 plugin to remove them.

"""
NO Patch plugin to prevent applying the patch again.

name: nopatch
version: 0.1.0
patches:
  openedx-dockerfile-git-patches-default: "#"
"""

from tutor import hooks

hooks.Filters.ENV_PATCHES.add_item(
  (
  "openedx-dockerfile-git-patches-default",
  "#"
  )
)

4.2.1. Creating and enabling the Plugin#

If you do not know what a Tutor plugin is, you can follow this quick guide.

You need to create a file called nopatch.py inside the plugins directory declared in 1.4. Create your Environment ($(pwd)/plugins for this guide)

Then you need to enable the plugin by running

tutor plugins enable nopatch

And save your configuration.

tutor config save

If you are using git, you will notice that the Dockerfile has changed to remove the patches.

4.2.2. Consequences#

This plugin will cause your images to never apply the patches inserted by Tutor, therefore you will have to manually check if there are any new patches every time you upgrade your Tutor version.

4.3. Update your Repo#

You should now commit and push your changes to your fork repo since the Tutor build commands use the remote repo rather than your local one.

cd $(pwd)/volumes/edx-platform
git add .
git commit -m "Your Commit Message"

4.4. Build your Image#

You can now build your image. Since you are using a custom edx-platform repository, you need to let Tutor know where to look.

Use the following command with added arguments to build your custom openedx image.

tutor images build openedx --build-arg EDX_PLATFORM_REPOSITORY="<https://github.com/><USERNAME>/edx-platform" --build-arg EDX_PLATFORM_VERSION="<USERNAME>-dev"

4.4.2. Private edx-platform Fork#

If you are using a private edx-platform fork, you can create a Personal access token ( https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and add it to the EDX_PLATFORM_REPOSITORY argument.

tutor images build openedx --build-arg EDX_PLATFORM_REPOSITORY="<https://<PAT>@github.com/><USERNAME>/edx-platform" --build-arg EDX_PLATFORM_VERSION="<USERNAME>-dev"

4.5. Use your Image in Production#

Simply follow 2.3. (Optional) Tag Your Images to change the image Tutor will use when you run tutor local quickstart.