Contributing
We follow a standard feature branch workflow, as described here.
Example development workflow
Setup:
# clone repository $ git clone https://github.com/Earth-Information-System/fireatlas.git $ cd fireatlas # create and activate environment named `fire_env` $ conda env create -f env.yml $ conda activate fire_env # install dev dependencies and editable install of fireatlas $ python -m pip install -e '.[dev]'Create a new branch and make your changes as needed. Name the branch {your initials}/{descriptive feature name}. Keep commits small and focused- one commit should contain only one logical set of related changes.
$ git switch -c zb/add-noaa21 # make changes $ git add fireatlas/my_changed_file.py $ git commit -m "Add NOAA21 inputs"Run tests from fireatlas root dir (requires optional dev dependencies from step 1)
$ python -m pytest -v tests/OPTIONAL: run long-running (~15mins) integration tests
$ python -m pytest -v --runslow tests/Push to your branch on GitHub.
# The first time you push your new branch, set upstream: $ git push -u origin zb/add-noaa21 # Subsequently, you can just do: $ git pushWhen you are ready, create a pull request to merge your branch into the main branch,
main. Most substantive changes should be reviewed by another team member, and all changes must wait to be merged until the automated tests have passed.
Developing with Jupyter notebooks
To use a Jupyter Notebook with the development environment:
# First, create conda environment following directions in Step 1 above
# Then, from the fireatlasroot dir:
$ python -m pip install '.[dev]'
$ python -m ipykernel install --user --name fire_env --display-name "Python - FEDS development environment"
# wait 30 seconds or so, then launch a new notebook. You should see this environment as an option to choose from in the launcher.
This assumes you are using a JupyterLab interface such as the MAAP ADE or MAAP Hub.
Environment and dependency management in depth
The FEDS codebase is used in a few different contexts- for development locally or on the MAAP Hub, within the MAAP DPS production pipeline for NRT data processing, in the CI infrastructure for testing, and occasionally as a library (e.g. if an analysis notebook needs to pull in a function from postprocessing). This creates a little additional complexity for environment management.
You may notice that some dependencies are listed in both pyproject.toml and env.yml. This is intentional. pyproject.toml defines fireatlas’s runtime dependencies, which is important if it is being used as a library. But, we actually use conda (env.yml) to install geospatial (e.g. geopandas) and numerical (e.g. numpy) libraries that have compiled dependencies such as GDAL from conda-forge binaries. In practice, this means that for environments we manage, conda installs the binary first, then pip sees that requirement is already satisfied and skips it.
As a rule of thumb- if you find that you need to pin a dependency (e.g., a new major version of pandas creates a breaking change), make sure to reflect that pin in both files. If you add a dependency with compiled components, add it to both. If you add a pure python dependency, only add it to pyproject.toml.
Conda environment files
We have two: env.yml and env.lock.yml. env.yml is the loose, human edited spec that is only version pinned when we have identified a real, breaking requirement. env.lock.yml is machine generated, with fully pinned exact versions for each package. Never edit this by hand- it defines the exact reproducible environment that DPS builds new production images from. It is platform specific, so only modify it on MAAP Hub or another Linux machine. Also, make sure that when you do this, the only dependencies present in your conda environment are those from env.yml and that you don’t accidentally add packages that you installed manually for different reasons or that were inherited from a base conda env.
Regenerate and commit to git env.lock.yml whenever you change the dependencies in env.yml or pyproject.toml (after making sure the environment still works, of course).
# should run in a conda env setup with:
conda remove -n fire_env -y # start clean
conda env create -f env.yml
conda activate fire_env
python -m pip install -e '.[prod]'
# platform specific, only run on Linux (e.g. the MAAP Hub)
# the grep drops fireatlas's own install line (see note below)
conda env export --no-builds | grep -v "fireatlas" > env.lock.yml
Note that fireatlas itself is deliberately not pinned in the lock. The lockfile captures just dependencies; then, fireatlas is installed using the version of the codebase available locally (pip install -e '.[dev]' for development, pip install --no-deps . in the DPS build).
Environment contexts
- Local development / MAAP Hub: A looser/lightweight environment resolution is fine for development work and, unlike the lockfile, works across different platforms (e.g. Mac, Linux).
conda env create -f env.yml # set up env
conda activate fire_env
python -m pip install -e '.[dev]'
MAAP DPS images and GH Actions CI: DPS production environments are built by
maap_runtime/run_dps_build.shinside the MAAP base image whenever we register an algorithm with DPS. This uses the pinnedenv.lock.yml, as does the GH Actions CI testing, so that DPS resolves exactly the same environment as was tested. This way, we can always have a reproducible build and won’t get silently bitten if an upstream library introduces a breaking dependency conflict with another of our dependencies.Installing
fireatlasas a library in another project:pyproject.tomldoes provide a complete dependency contract for importingfireatlasas a library in other projects/environments, for example for a paper analysis. In this case, you can install it directly with pip and let it resolve everything with the existing environment, not using conda at all. This is good for playing nicely with version requirements that the other project may have and allowing the user to manage their environment more freely without causing conflicts via strict pinned requirements in our library.
pip install "fireatlas @ git+https://github.com/Earth-Information-System/fireatlas.git"
Editing Docs
We use Quarto to build this website. Most of the source files are plaintext .qmd files written in Quarto markdown. You can edit them in any text editor, including with the MAAP ADE or even directly on GitHub. However, if you are making more extensive changes or using formatting, you should preview what the rendered website will look like.
Website formatting and page structure are defined in _quarto.yml.
You can use VSCode, Jupyter, RStudio, NeoVim, or any text editor + CLI to run Quarto. Info on getting started can be found in the Quarto docs here.
Below is a quick demo of how you can use VSCode on your local machine to preview the website and edit with a visual (WYSIWYG) editor.
Install jupyter if needed (
pip install jupyter)Install Quarto extension in VSCode
Clone this repo and open it in VSCode.
Make edits to
.qmd,.mdor.ipynbfiles, or_quarto.yml.TipOpen the Command Palette (cmd-shift-p on Mac) and select
Quarto: Edit in Visual Modefor What You See Is What You Get editing. SelectQuarto: Edit in Source Modeto go back.Preview website. Select the Preview icon in upper right, use the keyboard shortcut cmd-shift-k, or use the
Quarto: Previewcommand from the Command Palette.Once everything looks good, commit your changes and push to GitHub. If you pushed directly to main, GitHub will automatically build and deploy the new website reflecting your changes. If you are working on a branch, GitHub will build and deploy your changes when you create a pull request and merge your branch into main.
Most users should never need to do this. However: to rebuild the active GitHub Pages website from a branch other than the main branch, checkout that branch and run quarto publish gh-pages from within that directory. This will trigger a site rebuild based on the checked out branch. Note that it will be superseded as soon as another push is made to the main branch.