Add magnetic orderings workflow #432

mattmcdermott · 2023-07-11T19:58:08Z

Summary

This PR includes the addition of a flow and builder for collinear magnetic ordering enumeration/calculations. This flow is based on the original MagneticOrderingsWF in atomate1. This extends the first sketch by @mkhorton in #39.

The final postprocessing step is included as a job and as a builder.

Most of the workflow logic is written within common. In this PR I have implemented everything with VASP.

TODO (if any)

Finalize and merge PR into pymatgen for broken magnetic code Fix MagOrderingTransformation by enforcing implicit-zero magnetic moments in SpaceGroupAnalyzer pymatgen#3070
Fix possible bugs related to orderings changing between static and relax
Explore moving even more code into common (especially builder code)
Add tests

Checklist

Code is in the standard Python style.
The easiest way to handle this is to run the following in the correct sequence on
your local machine. Start with running black on your new code. This will
automatically reformat your code to PEP8 conventions and removes most issues. Then run
ruff.
Docstrings have been added in the Numpy docstring format.
Run ruff on your code.
Type annotations are highly encouraged. Run mypy to
type check your code.
Tests have been added for any new functionality or bug fixes.
All linting and tests pass.

codecov · 2023-07-11T20:13:22Z

Codecov Report

Attention: Patch coverage is 69.89247% with 84 lines in your changes are missing coverage. Please review.

Project coverage is 74.71%. Comparing base (29a5731) to head (e117f6f).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #432      +/-   ##
==========================================
- Coverage   74.94%   74.71%   -0.24%     
==========================================
  Files         136      140       +4     
  Lines       10513    10792     +279     
  Branches     1643     1673      +30     
==========================================
+ Hits         7879     8063     +184     
- Misses       2143     2229      +86     
- Partials      491      500       +9

Files	Coverage Δ
src/atomate2/common/schemas/magnetism.py	`96.85% <96.85%> (ø)`
src/atomate2/common/jobs/magnetism.py	`89.13% <89.13%> (ø)`
src/atomate2/common/flows/magnetism.py	`77.50% <77.50%> (ø)`
src/atomate2/common/builders/magnetism.py	`0.00% <0.00%> (ø)`

... and 1 file with indirect coverage changes

mkhorton · 2023-07-11T23:44:41Z

Amazing! I will review but don't expect major comments.

mattmcdermott · 2023-07-11T23:54:18Z

Thanks! Yes feedback especially on schemas would be useful -- I did base them on your previous sketches though.

mattmcdermott · 2023-07-12T20:30:03Z

FYI: I moved the builder into common. Of course, this opens up the possibility of the builder mixing outputs from different DFT codes; I addressed this by adding a property that is implemented in child classes. This property, _dft_code_query, is tacked onto the get_items query to ensure that only calcs from one DFT code are used.

Since this is the first builder in common, maybe we can standardize an approach like this for future common builders?

JaGeo · 2023-07-12T20:31:59Z

Approach for common sounds very good!

mkhorton

Great PR! Approved by me, although noting I'm not the maintainer here :) I added some comments mostly for context/gotchas.

mkhorton · 2023-07-19T20:23:33Z

src/atomate2/common/builders/magnetism.py

+        for group in grouped_tasks:
+            group_parent_structure = task["metadata"]["parent_structure"]
+
+            # parent structure should really be exactly identical (from same workflow)


I like this function. I wonder if we might benefit from the same code but in an exact_match() utility function somewhere instead, it seems like it might be useful more generally.

mkhorton · 2023-07-19T20:26:08Z

src/atomate2/common/schemas/magnetism.py

+            "as defined in pymatgen.analysis.magnetism.analyzer."
+        ),
+    )
+    magmoms: list[float] = Field(None, description="Magnetic moments of the structure.")


Not a request to modify, but adding a note for reference: it would be useful to round magnetic moments somewhat, since getting a primitive structure (with respect to the magnetic moments, i.e. without losing magnetic order) will not work as intended unless equivalent moments are exactly equal. Typically, the noise in the moment is sufficient such that this is not the case. I did not get around to adding this however.

Should we enable round_magmoms=True in the CollinearMagneticStructureAnalyzer?

Yes, I'd do that. Eventually the underlying algorithm in round_magmoms should be updated however. Simple rounding (e.g. np.round) will sometimes round two similar magmoms in opposite directions. I tried instead doing a KDE of the magmoms with a suitable Gaussian width and taking the peak positions, but this felt like it was over-complicating things!

mkhorton · 2023-07-19T20:27:42Z

src/atomate2/common/schemas/magnetism.py

+    relax_output: MagneticOrderingRelaxation | None = Field(
+        None, description="Relaxation output, if relaxation performed."
+    )
+    energy_diff_relax_static: str | float | None = Field(


Might be worth noting this field was only really present for the purpose of benchmarking (i.e., is the static calculation required).

Since I wrote the original workflow, I have since concluded that the static step should--at minimum--have a much smaller EDIFF, however I have not benchmarked this. The original benchmark results speak for themselves, however the difference in energies between orderings is sometimes very very small, such that I would have more confidence if we had a tighter tolerance.

This is an important (and still unresolved) point. Right now, the default static calculations in the workflow are made with the VASP StaticMaker, which I believe is EDIFF=1e-5.

The default StaticMaker (EDIFF=1e-5) is looser than TightRelaxMaker (ENCUT=700, EDIFFG=1e-7). Maybe we could use that parameter and this energy_diff_relax_static term to benchmark and see if we need to improve the EDIFF in StaticMaker.

mkhorton · 2023-07-19T20:28:33Z

src/atomate2/vasp/builders/magnetism.py

+)
+
+
+class MagneticOrderingsBuilder(MagneticOrderingsBuilderBase):


It's really nice to see how few additions were required to make this work with VASP. I wonder if this is a pattern that we should use elsewhere in atomate2? (If it's not already?)

I think it's a nice pattern, although it was tricky to figure out. I hope that this addition will be a nice example for future workflows :)

Looks nice and concise! We can follow this good example to migrate the current ElasticBuilder under vasp to common.

Thanks @Zhuoying, agreed :)

This would be great! I think we could then easily connect the forcefields as well 😃

mkhorton · 2023-07-19T20:29:39Z

src/atomate2/vasp/flows/magnetism.py

+        relaxations will be skipped (i.e., only static calculations).
+    default_magmoms : dict | None
+        Optional default mapping of magnetic elements to their initial magnetic moments
+        in µB. Generally these are chosen to be high-spin, since they can relax to a


One thing that might be useful to verify is the default spin on Co. Since the original workflow was written, the pymatgen default changed to low-spin Co, since unlike other elements, Co seemed happier to be trapped in a high-spin configuration even if less stable. Strictly, I think Co should probably be run twice with both configurations.

mattmcdermott · 2023-08-04T00:18:17Z

Btw I haven't forgotten about this; just a little preoccupied. Need to address a bug still and write tests!

mattmcdermott · 2023-09-04T20:18:07Z

Bug was addressed. See Issue #505 where AFM orderings were being destroyed between relax and static calcs. Fix was provided in PR #506.

Still need to write tests and then we should be about ready to merge!

mattmcdermott · 2023-09-05T23:19:55Z

@utf I just implemented your suggestion in #466. This workflow now contains a postprocessing job that shares much of the doc-constructing code used in the builder.

In addition to feedback about the postprocessing implementation, your feedback on how this is implemented within common would be very helpful, as this workflow might set the precedence for future additions to atomate2.

I'll be working on tests in the meantime and will let you know when they're ready :)

mattmcdermott · 2023-09-11T19:34:49Z

Test for magnetic ordering workflow has been added (using the developer guide). Since writing this code, @mcgalcode and I have also run the workflow and builder on 50+ structures, and it seems stable overall.

@utf Is there anything else I should add? It would be great if you could review Matt H's comments above as well; one discussion point is whether we should make the static jobs have a smaller EDIFF by default (e.g., 1e-6 or 1e-7). Instead, we could leave it the way it is (i.e., default StaticMaker) and let the user change this if desired.

I see now that the new test is failing on GitHub Actions because enumlib is a required dependency to enumerate magnetic orderings. What do you think we should do here?

mkhorton · 2023-10-04T06:44:28Z

have also run the workflow and builder on 50+ structures

This seems more than sufficient to approve. Would recommend leaving the EDIFF to the current atomate2 static default unless benchmarking suggests a tighter tolerance is required (although I agree it might help).

Re. enumlib, it's an easy compile but it's not consistently versioned. The one on conda-forge does seem like it was last updated 9 months ago, so fairly recent--perhaps we could try installing this in CI?

utf · 2023-10-12T08:47:05Z

@Zhuoying please can you take a look at this PR and the comments.

@mattmcdermott once @Zhuoying has had a look over, I will do a final pass.

Regarding enumlib, I think we can follow @mkhorton's advice and try and install the one on conda forge. Longer term, it would be great if we could replace enumlib with a Python only package like bsym

Zhuoying

Nice structured workflow. Can be used as a good example for new developers when they need a standardized postprocessing. The remaining issue might be:
(i) Is the default incar settings (EDIFF=1e-5) in StaticMaker sufficient (may need some benchmark evidence).
(ii) Double-check Co default spin (default changed to low-spin in pymatgen)
(iii) Pydantic v2 update (if any)
Thanks @mattmcdermott for the excellent PR.

src/atomate2/common/flows/magnetism.py

src/atomate2/common/schemas/magnetism.py

Zhuoying · 2023-10-16T23:57:56Z

src/atomate2/vasp/builders/magnetism.py

+)
+
+
+class MagneticOrderingsBuilder(MagneticOrderingsBuilderBase):


Looks nice and concise! We can follow this good example to migrate the current ElasticBuilder under vasp to common.

Zhuoying · 2023-10-17T00:05:34Z

src/atomate2/common/schemas/magnetism.py

+    relax_output: MagneticOrderingRelaxation | None = Field(
+        None, description="Relaxation output, if relaxation performed."
+    )
+    energy_diff_relax_static: str | float | None = Field(


The default StaticMaker (EDIFF=1e-5) is looser than TightRelaxMaker (ENCUT=700, EDIFFG=1e-7). Maybe we could use that parameter and this energy_diff_relax_static term to benchmark and see if we need to improve the EDIFF in StaticMaker.

src/atomate2/vasp/flows/magnetism.py

mattmcdermott · 2023-10-18T20:20:55Z

Hi @Zhuoying, thanks for the nice feedback and code review; this is very helpful!

The remaining issue might be: (i) Is the default incar settings (EDIFF=1e-5) in StaticMaker sufficient (may need some benchmark evidence).

I do think EDIFF for the static calc should be stricter by default, per @mkhorton's recommendation. So far, I have run relax+static calcs on 285 different orderings (from 26 structures) with the current defaults (EDIFF=1e-5). I made a histogram of all their (normalized) energy_diff_relax_static values.

The mean difference is -0.0016 eV/atom (i.e. the static calculation tends to be that much lower in energy than the relax), but there are some outliers! I don't have a good sense on what this means with regards to picking an EDIFF, but I imagine this supports the idea that we should be stricter... Thoughts?

(ii) Double-check Co default spin (default changed to low-spin in pymatgen)

While the default Co is now low-spin, the MagneticStructureEnumerator still uses the older default magnetic moments by loading them from file here:

https://github.com/materialsproject/pymatgen/blob/cfd47add8958af795dabcfe1661cf8890ad56415/pymatgen/analysis/magnetism/default_magmoms.yaml#L1-L4

This means that right now the default for the magnetic orderings workflow will be to initialize Co to high-spin. @mkhorton recommended running both low-spin and high-spin. Is this something we should implement in pymatgen or implement here?

(iii) Pydantic v2 update (if any)

Will look into!

JaGeo · 2023-10-18T20:24:07Z

I do think EDIFF for the static calc should be stricter by default, per @mkhorton's recommendation. So far, I have run relax+static calcs on 285 different orderings (from 26 structures) with the current defaults (EDIFF=1e-5). I made a histogram of all their (normalized) energy_diff_relax_static values.

The mean difference is -0.0016 eV/atom (i.e. the static calculation tends to be that much lower in energy than the relax), but there are some outliers! I don't have a good sense on what this means with regards to picking an EDIFF, but I imagine this supports the idea that we should be stricter... Thoughts?

1e-7 or at least 1e-6. 1e-7 is usually my default for manual phonon runs. I also implemented a phonon static run that could be used (will be moved from common to vasp in one of the open pull requests):

atomate2/src/atomate2/common/jobs/phonons.py

Line 355 in 2843d04

"EDIFF": 1e-7,

JaGeo · 2023-10-19T06:25:39Z

Just to shortly add to my last comment, @mattmcdermott : I think one would need to compare the energies resulting from two static runs. comparing the optimization with a static run could lead to wrong conclusions as the volume might change during the optimization which has an influence on the basis set quality and therefore the energy from the optimization run.

Zhuoying · 2023-10-24T23:43:07Z

@mattmcdermott Thanks for providing the benchmarking histogram! It gives some quantitative evidence on whether we should choose a more strict EDIFF.

Two follow-up comments:
(i) EDIFF in static_maker: From the histogram, there are more than 10% of orderings (~30/285) Ediff >10 meV/atom. If the difference in energies between orderings is sometimes very very small (<5meV/atom), the current default (1e-5) needs to be modified to a smaller value (i.e., 1e-7) rather than leaving it for users to decide IMO.

(ii) relax_maker=None: I suppose when you have confidence in your input structure (well-relaxed), you can turn the relax_maker off. I have not come up with a situation where you can do this. -If magnetism for different orderings is set well and done with relaxation, it sounds like manually doing the first half of the workflow. If not and skip the relaxation step, I might be worried about the accuracy since magnetism does matter for the optimized structures. So if you have any idea when we can turn it off, please let me know. Otherwise, we might consider adding a warning for relax_maker=None and turning it on as the default?

@utf The discussions and replies to my review are done. Now it is ready for your final review.

mkhorton · 2023-10-25T00:25:34Z

This means that right now the default for the magnetic orderings workflow will be to initialize Co to high-spin. @mkhorton recommended running both low-spin and high-spin. Is this something we should implement in pymatgen or implement here?

@mattmcdermott This is a personal judgement call. Co has historically been tricky, and seems more likely to get stuck in a high-spin configuration than other elements. The decision to switch to a low-spin by default was simply because it seemed more common (i.e., what would be the most sensible default value?) but of course plenty of high-spin Co materials exist, and this is all anecdotal. The most exhaustive option would be to consider multiple states for all elements, but this would likely be quite inefficient.

For the purposes of this workflow, if you can predict the oxidation state ahead of time, that's helpful, and the default magnetic moment can be set differently according to that.

I would not recommend any changes to this workflow at this time, there isn't enough evidence either way (that I know of). At the end of the day, porting this workflow to atomate2 at all is super helpful, and it doesn't mean further improvements can't be made later.

Modifying the workflow/enumerator to allow multiple spin states per element would be an option, but I think this might be an awkward change. Perhaps the best option is simply giving the user a warning if they're calculating a material including Co and suggest they attempt both?

mattmcdermott · 2023-10-27T17:46:00Z

@Zhuoying and @JaGeo Thank you for the helpful discussion on EDIFF values. I agree with your assessment and will implement EDIFF=1e-7 as the new default.
@Zhuoying Regarding the optional relaxation -- note that the VASP-implemented workflow includes a RelaxMaker by default (it only defaults to None in the common flow). I will add a warning when None is supplied, though.
@mkhorton Providing the warning you described for Co-containing structures seems like the best solution for now.

Thanks all for your help. I will update the PR today, and then we should be good for final review by @utf

mattmcdermott · 2023-10-28T00:36:38Z

I added support for enumlib installation during testing with Github Actions per the commands used in the pymatgen github workflows. Adds about 30s to the testing run; hopefully this is okay!

fraricci · 2023-10-28T18:09:09Z

If I can chime in here. I just posted a bug issue in pmg which is affecting the atomate MagWF. materialsproject/pymatgen#3431

It could be useful to test if that bug affects the current MagWF in atomate2 as well.

utf · 2023-11-16T10:11:54Z

This looks fantastic. Thanks so much @mattmcdermott. Once tests pass I'm happy to merge.

mattmcdermott · 2024-03-31T17:02:22Z

@utf the magnetic orderings workflow is now implemented only in common, with defaults for VASP and a note about ensuring compatible TaskDoc implementation for alternate DFT codes.

Note I bumped the emmet-core version to include the TaskDoc change I made there (materialsproject/emmet#945)

The failing tests are from other modules.

No rush to pull but let me know if you are envisioning any other changes!

utf · 2024-04-03T10:39:07Z

This is fantastic. Thanks very much!

The tests are failing because of a recent change in emmet. I can merge this once materialsproject/emmet#974 is merged and a new version released.

munrojm · 2024-04-03T16:38:29Z

Just merged, it should be released soon.

JaGeo · 2024-05-11T21:02:27Z

@utf I am merging as now all tests pass and you said it is good to merge as soon as the emmet version is fixed (only codecov slightly decreased now)

@mattmcdermott

* Initial sketch of `MagneticOrderingsMaker` * Add schema to match original atomate `MagneticOrderingsToDb` format * move magnetism jobs to common * move magnetism flows to common * clean up magnetic ordering documentation/typing * make "prev calc dir" a code-dependent argument name for static maker * clean up typing, imports, docstrings * begin porting over magnetic workflow analysis code / schemas * fix jobflow logic for magnetic enumerator flow * rework for loop for orderings * migrate magnetic orderings wf to builder format * Improve naming * remove output storing * add magnetic ordering builder implementation * remove unused imports in magnetism jobs * fix bug in task creation in magnetism builder * update magnetic ordering job documentation * make sure name "magnetic orderings" is consistent * make sure name "magnetic orderings" is consistent pt. 2 * move magnetic orderings builder into common * ensure copying of magmoms between relax and static; linting * fix copy magmom * fix kwarg location (mistake previously) * add final False to copy vasp magmoms * debug test of magmoms * fix bug in magmoms * delete unused line * move magnetic ordering & symmetry comparison logic to common * linting * linting pt 2 * add from_tasks() support; extract logic to common * add and fix postprocessing job * linting * fix error in structure matching in builder It's actually dangerous to match by lattice/site matrices; sometimes the structures are not identically defined despite being the same structure * add test for magnetic ordering workflow * Add warning if relax_maker not provided; stricter EDIFF settings * Add enumlib install and Cobalt warning * move Co warning * add Optional to magnetism schemas * ignore enumlib's tests * make testing directory explicit * regenerate test data for mag_ordering workflow * fix broken test (due to new EDIFF=1e-7 data) * remove the patch fix we had for prev_dir now that atomate2 is updated * fix breaking python 3.8 test * move imports and fix typing problem * try making tests dir not explicit * change dir in github action to move enumlib outside testing scope * update @mattmcdermott contributor info * attempt to fix coverage CI bug * monty decoding for magnetism builder * fully implement magnetic_orderings workflow into common * add warning about TaskDoc * bump emmet-core version * clean up magnetic ordering wf files * bump emmet-core again --------- Co-authored-by: Matthew Horton <[email protected]> Co-authored-by: J. George <[email protected]>

@mattmcdermott

* Initial sketch of `MagneticOrderingsMaker` * Add schema to match original atomate `MagneticOrderingsToDb` format * move magnetism jobs to common * move magnetism flows to common * clean up magnetic ordering documentation/typing * make "prev calc dir" a code-dependent argument name for static maker * clean up typing, imports, docstrings * begin porting over magnetic workflow analysis code / schemas * fix jobflow logic for magnetic enumerator flow * rework for loop for orderings * migrate magnetic orderings wf to builder format * Improve naming * remove output storing * add magnetic ordering builder implementation * remove unused imports in magnetism jobs * fix bug in task creation in magnetism builder * update magnetic ordering job documentation * make sure name "magnetic orderings" is consistent * make sure name "magnetic orderings" is consistent pt. 2 * move magnetic orderings builder into common * ensure copying of magmoms between relax and static; linting * fix copy magmom * fix kwarg location (mistake previously) * add final False to copy vasp magmoms * debug test of magmoms * fix bug in magmoms * delete unused line * move magnetic ordering & symmetry comparison logic to common * linting * linting pt 2 * add from_tasks() support; extract logic to common * add and fix postprocessing job * linting * fix error in structure matching in builder It's actually dangerous to match by lattice/site matrices; sometimes the structures are not identically defined despite being the same structure * add test for magnetic ordering workflow * Add warning if relax_maker not provided; stricter EDIFF settings * Add enumlib install and Cobalt warning * move Co warning * add Optional to magnetism schemas * ignore enumlib's tests * make testing directory explicit * regenerate test data for mag_ordering workflow * fix broken test (due to new EDIFF=1e-7 data) * remove the patch fix we had for prev_dir now that atomate2 is updated * fix breaking python 3.8 test * move imports and fix typing problem * try making tests dir not explicit * change dir in github action to move enumlib outside testing scope * update @mattmcdermott contributor info * attempt to fix coverage CI bug * monty decoding for magnetism builder * fully implement magnetic_orderings workflow into common * add warning about TaskDoc * bump emmet-core version * clean up magnetic ordering wf files * bump emmet-core again --------- Co-authored-by: Matthew Horton <[email protected]> Co-authored-by: J. George <[email protected]>

mkhorton approved these changes Jul 19, 2023

View reviewed changes

mattmcdermott mentioned this pull request Aug 9, 2023

Standardize approach for postprocessing data: use builders? #466

Open

mattmcdermott changed the title ~~[WIP] Magnetic orderings workflow~~ Magnetic orderings workflow Sep 11, 2023

mattmcdermott changed the title ~~Magnetic orderings workflow~~ Add magnetic orderings workflow Sep 11, 2023

utf mentioned this pull request Oct 12, 2023

Add FHI-aims DFT calculator #562

Merged

5 tasks

Zhuoying self-requested a review October 13, 2023 04:18

Zhuoying reviewed Oct 17, 2023

View reviewed changes

Zhuoying mentioned this pull request Oct 17, 2023

[WIP] Initial sketch of MagneticOrderingsMaker #39

Closed

9 tasks

janosh force-pushed the main branch from 6ceb1ec to 3764841 Compare October 27, 2023 00:12

fraricci mentioned this pull request Oct 28, 2023

Bug: MPSet/MagneticStructureEnumerator: different Magmoms/Spins in INCAR than those in the Structure/POSCAR materialsproject/pymatgen#3431

Open

mattmcdermott added 20 commits March 31, 2024 09:39

Add enumlib install and Cobalt warning

f2969da

move Co warning

e94c94d

add Optional to magnetism schemas

5c2ad05

ignore enumlib's tests

3bc98de

make testing directory explicit

5b510b2

regenerate test data for mag_ordering workflow

279399f

fix broken test (due to new EDIFF=1e-7 data)

3dbb48e

remove the patch fix we had for prev_dir now that atomate2 is updated

dee7488

fix breaking python 3.8 test

48707bf

move imports and fix typing problem

ad20f0a

try making tests dir not explicit

816ee34

change dir in github action to move enumlib outside testing scope

557ba32

update @mattmcdermott contributor info

ba322e8

attempt to fix coverage CI bug

05bff18

monty decoding for magnetism builder

30240ae

fully implement magnetic_orderings workflow into common

3265dd3

add warning about TaskDoc

71527a8

bump emmet-core version

5b8f295

clean up magnetic ordering wf files

bb9ed53

bump emmet-core again

fe751cf

mattmcdermott force-pushed the magnetic_orderings branch from 882fa27 to fe751cf Compare March 31, 2024 16:40

JaGeo added 2 commits May 11, 2024 22:15

Merge branch 'main' into magnetic_orderings

3adf290

Merge branch 'main' into magnetic_orderings

e117f6f

JaGeo merged commit 3b550be into materialsproject:main May 11, 2024
6 of 7 checks passed

		)


		class MagneticOrderingsBuilder(MagneticOrderingsBuilderBase):

Add magnetic orderings workflow #432

Add magnetic orderings workflow #432

Conversation

mattmcdermott commented Jul 11, 2023 • edited Loading

Summary

TODO (if any)

Checklist

codecov bot commented Jul 11, 2023 • edited Loading

Codecov Report

mkhorton commented Jul 11, 2023

mattmcdermott commented Jul 11, 2023

mattmcdermott commented Jul 12, 2023

JaGeo commented Jul 12, 2023

mkhorton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmcdermott Sep 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmcdermott commented Aug 4, 2023

mattmcdermott commented Sep 4, 2023

mattmcdermott commented Sep 5, 2023

mattmcdermott commented Sep 11, 2023

mkhorton commented Oct 4, 2023 • edited Loading

utf commented Oct 12, 2023 • edited Loading

Zhuoying left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmcdermott commented Oct 18, 2023

JaGeo commented Oct 18, 2023 • edited Loading

JaGeo commented Oct 19, 2023 • edited Loading

Zhuoying commented Oct 24, 2023

mkhorton commented Oct 25, 2023

mattmcdermott commented Oct 27, 2023

mattmcdermott commented Oct 28, 2023

fraricci commented Oct 28, 2023

utf commented Nov 16, 2023

mattmcdermott commented Mar 31, 2024 • edited Loading

utf commented Apr 3, 2024

munrojm commented Apr 3, 2024

JaGeo commented May 11, 2024

mattmcdermott commented Jul 11, 2023 •

edited

Loading

codecov bot commented Jul 11, 2023 •

edited

Loading

mattmcdermott Sep 11, 2023 •

edited

Loading

mkhorton commented Oct 4, 2023 •

edited

Loading

utf commented Oct 12, 2023 •

edited

Loading

Zhuoying left a comment •

edited

Loading

JaGeo commented Oct 18, 2023 •

edited

Loading

JaGeo commented Oct 19, 2023 •

edited

Loading

mattmcdermott commented Mar 31, 2024 •

edited

Loading