Mini Course Docs: Tutor Fork Quickstart with `eduNEXT/tvm `__ ============================================================================================= Latest update: May 23, 2022 6:06 PM (Please change the versions accordingly if this post is too old.) Table of Contents ----------------- .. contents:: Steps :class: no-bullets 0. Prerequisites ---------------- - https://github.com/pyenv/pyenv - https://github.com/pyenv/pyenv-virtualenv 1. Set up Environment --------------------- Collect the versions for the software you will be using. .. dropdown:: For example, my current setup for Maple uses the following. **Tutor:** 13.2.2 **Python:** 3.8.12 **edx-platform:** https://github.com/overhangio/tutor/blob/v13.2.2/tutor/templates/config/defaults.yml#L51 **TVM:** 1.0.0 (Choose according to your release) 1.1. Create Working Directory ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a folder to store your Tutor environment. :: mkdir tutor-maple cd tutor-maple The following examples assume you are running the code from the ``~/tutor-maple`` directory. I will also refer to this directory as ``$(pwd)``. You should replace ``$(pwd)`` with the result of running the following code in your working directory. :: pwd 1.2. Install TVM ~~~~~~~~~~~~~~~~ You should now install the TVM package. .. code:: bash pip install git+https://github.com/eduNEXT/tvm.git@v1.0.0 1.3. Install Tutor ~~~~~~~~~~~~~~~~~~ The TVM enables you to install Tutor with a simple command. :: tvm install v13.2.2 1.4. Create your Environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can now use the Environment Manager function of ``tvm`` to create your work environment. .. code:: bash tvm project init 1.5. Enable your Environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You need to enable the environment after creating it. .. code:: bash source .tvm/bin/activate 1.6. Install Required Plugins ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You need to install the MFE plugin at this point for the installation to succeed. .. code:: bash tvm plugins install mfe You can install additional Python plugins by running the ``tvm plugins install`` command with the plugin name. The list of official plugins are here: https://overhang.io/tutor/plugins 1.7. Save Configuration ~~~~~~~~~~~~~~~~~~~~~~~ You should run the save configuration command now to generate the Tutor environment at your working directory. We do not want to start Tutor yet, so we will only create the configuration files. .. code:: bash tutor config save --interactive You should answer first question as **no** since you will be doing development. 1.7.1. (Optional) Initialize Git Repository ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can initialize a git repository inside your working directory at this point to store your Tutor configuration. **You should be careful about uploading this git repository as it will include all your database passwords and other secretes. This repository should be a ``private`` repo if you have to upload it.** .. code:: bash git init 1.7.2. (Optional) .gitignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You might want to create a .gitignore file, so that you do not include the Tutor data files. .. code:: bash volumes data* .tvm 1.8. Additional Tutor Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You should add the following values to the ``config.yml`` located inside your Tutor environment created in :ref:`1.4. Create your Environment` . (You can find it by running ``echo $(pwd)`` if you lost it.) :: LOCAL_PROJECT_NAME: tutor_maple DEV_PROJECT_NAME: tutor_maple_dev Your Tutor installation will be designated the name ``tutor`` in your Docker environment by default, if you do not change it here, you will overwrite that Docker environment. Do not forget to save your ``config.yml`` file by running ``tutor config save``. 2. Build Development Image -------------------------- You are now ready to create the ``openedx`` and ``openedx Development`` images. 2.1. Open edX Image ~~~~~~~~~~~~~~~~~~~~ You can now run the familiar **quickstart** command. .. code:: bash tutor local quickstart Once you confirm that your site is up and running, stop it. .. code:: bash tutor local stop You have now downloaded the openedx image and are ready to build the development image. 2.2. Open edX Development Image ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The development image is created by simply running the following code. .. code:: bash tutor dev dc build lms After the image is build, you can start your development instance in the detached mode. .. code:: bash tutor dev start -d Once you confirmed that your development site is up and running, stop it. .. code:: bash tutor dev stop 2.3. (Optional) Tag Your Images ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You should now tag your images to upload them to your image repository if you will be using different machines for development and production. :: DOCKER_IMAGE_OPENEDX: /: DOCKER_IMAGE_OPENEDX_DEV: /: You should now tag your development image. 2.3.1. (Optional) Create Development Image Again to Tag it ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (You can also find the image ID from Docker Desktop and use the Docker CLI to this step.) .. code:: bash tutor dev dc build lms 2.3.2 (Optional) Upload Your Image ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can run the following commands to upload your images. .. code:: bash tutor images push openedx docker push $(tutor config printvalue DOCKER_IMAGE_OPENEDX_DEV) 1. Creating Development Environment ----------------------------------- You are now ready to start working on your own ``edx-platform`` fork. 3.1. Create Your Fork ~~~~~~~~~~~~~~~~~~~~~ Use GitHub to create a fork of the **``edx-platform``**. https://github.com/openedx/edx-platform 3.2. Download Your Forked Repo to Your Development Environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You are now ready to use your forked repo for development. First receive the files from the Tutor installation. .. code:: bash mkdir -p $(tutor config printroot)/volumes/edx-platform tutor dev copyfrom lms /openedx/edx-platform $(tutor config printroot)/volumes/ This way you will be able to use the development tools that are installed inside the Development image. It will take some time to copy over the files. Then change the remote to point to your fork. .. code:: bash cd $(pwd)/volumes/edx-platform git remote set-url origin Sync your repo. .. code:: bash git fetch Finally change to the branch you will be using. .. code:: bash git checkout -b -dev 3.3. Point Tutor Towards Your Forked Repo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You should create an override file ``docker-compose.override.yml``\ at this location ``$(pwd)/env/dev/docker-compose.override.yml`` with the following content. You should replace the variables with your working directory. .. code:: yaml version: "3.7" services: lms: volumes: - "/volumes/edx-platform:/openedx/edx-platform" cms: volumes: - "/volumes/edx-platform:/openedx/edx-platform" lms-worker: volumes: - "/volumes/edx-platform:/openedx/edx-platform" cms-worker: volumes: - "/volumes/edx-platform:/openedx/edx-platform" 3.4. Start Development ~~~~~~~~~~~~~~~~~~~~~~ Your Tutor installation will now use your fork. .. code:: bash tutor dev start Congratulations, you now have a development environment to start working on an ``edx-platform`` fork. 4. Build your Custom Image -------------------------- You have done your changes and are now ready to build your custom image. 4.1. Apply the Patches ~~~~~~~~~~~~~~~~~~~~~~ You need to make sure you have all the patches applied to your codebase because we will remove them to prevent problems. You can find the current patches from the Dockerfile at the ``$(pwd)/env/build/openedx/Dockerfile`` directory. The patches section will start with the following code. :: # Identify tutor user to cherry-pick commits RUN git config --global user.email "tutor@overhang.io" \\ && git config --global user.name "Tutor" # Patch edx-platform You can directly run the codes after the ``RUN`` command in your ``edx-platform`` fork root directory (which is ``$(pwd)/volumes/edx-platform`` if you use the one suggested in this guide). .. dropdown:: For example, Tutor 13.2.2 uses the following patches. :: # Fix forum notification for questions # RUN git fetch --depth=2 03731f19459e558f188c06aac5cc9ca1bbc675c2 && git cherry-pick 03731f19459e558f188c06aac5cc9ca1bbc675c2 # SAML security fix # RUN git fetch --depth=2 3b985f207853e88090d68a81acd52866b71f5af7 && git cherry-pick 3b985f207853e88090d68a81acd52866b71f5af7 # Rate limiting security fix # RUN git fetch --depth=2 b5723e416e628cac4fa84392ca13e1b72817674f && git cherry-pick b5723e416e628cac4fa84392ca13e1b72817674f # Fix studio-frontend by pinning the installed version # RUN git fetch --depth=2 53ea60eee86e094f35815ac1c4114d6811f4d458 && git cherry-pick 53ea60eee86e094f35815ac1c4114d6811f4d458 You simply need to run the following commands in your ``edx-platform`` root directory. .. code:: bash git fetch --depth=2 03731f19459e558f188c06aac5cc9ca1bbc675c2 && git cherry-pick 03731f19459e558f188c06aac5cc9ca1bbc675c2 git fetch --depth=2 3b985f207853e88090d68a81acd52866b71f5af7 && git cherry-pick 3b985f207853e88090d68a81acd52866b71f5af7 git fetch --depth=2 b5723e416e628cac4fa84392ca13e1b72817674f && git cherry-pick b5723e416e628cac4fa84392ca13e1b72817674f git fetch --depth=2 53ea60eee86e094f35815ac1c4114d6811f4d458 && git cherry-pick 53ea60eee86e094f35815ac1c4114d6811f4d458 4.2. Remove the Patches ~~~~~~~~~~~~~~~~~~~~~~~ Now that you applied the patches, you do not need them anymore, use the following v1 plugin to remove them. .. code:: yaml """ NO Patch plugin to prevent applying the patch again. name: nopatch version: 0.1.0 patches: openedx-dockerfile-git-patches-default: "#" """ from tutor import hooks hooks.Filters.ENV_PATCHES.add_item( ( "openedx-dockerfile-git-patches-default", "#" ) ) 4.2.1. Creating and enabling the Plugin ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. dropdown:: If you do not know what a Tutor plugin is, you can follow this quick guide. You need to create a file called ``nopatch.py`` inside the plugins directory declared in :ref:`1.4. Create your Environment` (``$(pwd)/plugins`` for this guide) Then you need to enable the plugin by running .. code:: bash tutor plugins enable nopatch And save your configuration. .. code:: bash tutor config save If you are using git, you will notice that the Dockerfile has changed to remove the patches. 4.2.2. Consequences ^^^^^^^^^^^^^^^^^^^ This plugin will cause your images to never apply the patches inserted by Tutor, therefore you will have to manually check if there are any new patches every time you upgrade your Tutor version. 4.3. Update your Repo ~~~~~~~~~~~~~~~~~~~~~ You should now commit and push your changes to your fork repo since the Tutor build commands use the remote repo rather than your local one. .. code:: bash cd $(pwd)/volumes/edx-platform git add . git commit -m "Your Commit Message" 4.4. Build your Image ~~~~~~~~~~~~~~~~~~~~~ You can now build your image. Since you are using a custom ``edx-platform`` repository, you need to let Tutor know where to look. Use the following command with added arguments to build your custom ``openedx`` image. .. code:: bash tutor images build openedx --build-arg EDX_PLATFORM_REPOSITORY="/edx-platform" --build-arg EDX_PLATFORM_VERSION="-dev" 4.4.1. Tagging the Image ^^^^^^^^^^^^^^^^^^^^^^^^ Simply follow :ref:`2.3. (Optional) Tag Your Images` to tag your images. 4.4.2. Private ``edx-platform`` Fork ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you are using a private ``edx-platform`` fork, you can create a **Personal access token** ( https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and add it to the ``EDX_PLATFORM_REPOSITORY`` argument. .. code:: bash tutor images build openedx --build-arg EDX_PLATFORM_REPOSITORY="@github.com/>/edx-platform" --build-arg EDX_PLATFORM_VERSION="-dev" 4.5. Use your Image in Production ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Simply follow :ref:`2.3. (Optional) Tag Your Images` to change the image Tutor will use when you run ``tutor local quickstart``.