-
-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[REVIEW]: retworkx: A High-Performance Graph Library for Python #3968
Comments
Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @szhorvat, @inakleinbottle it looks like you're currently assigned to review this paper 🎉. Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post. ⭐ Important ⭐ If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿 To fix this do the following two things:
For a list of things I can do to help you, just type:
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
|
PDF failed to compile for issue #3968 with the following error:
|
|
Since JOSS reviews are so different from the typical peer review, I'd like to let you know in advance how I plan to proceed, just to make sure we are all on the same page. Editors, authors, please let me know if I should change this approach. I will prefix the titles of issues I open in the retworkx repository with I will post things that concern only the paper (and not the library) directly here. The same with things that I believe concern the editors. I will make it clear here when I consider the review completed. I won't be able to do it in a single go. Instead, I expect to have an ongoing conversation between the reviewers and authors for a couple of weeks. I will sometimes wait for answers to my comments/questions before proceeding. This is a large library and it will take time to get to know it. It is the first time that I write such an introduction. I decided to do it because it has happened in the past that people seemed to have differing concepts about how reviews are carried out, and communication wasn't as effective as it could have been. |
And thus we arrive to my first comment. This checklist item isn't fulfilled at the moment:
Can you please let me know how I can seek support? I expect I will have lots of questions which you may not want to see in the issue tracker (not bug reports or feature suggestions). Please also make sure that the proper support channels are clearly communicated to users. |
@whedon generate pdf from branch joss-paper |
|
We have a public slack channel as well as an IRC channel on oftc (the irc channel is empty except for me right now). I'll push a PR that adds a section to the bottom of the readme shortly that mentions these. |
I just remembered we do briefly mention the slack channel in our contributing documentation already: https://github.com/Qiskit/retworkx/blob/main/CONTRIBUTING.md#module-directories-when-a-single-file-is-not-enough (which also gets put in our published documentation: https://qiskit.org/documentation/retworkx/CONTRIBUTING.html?highlight=slack#module-directories-when-a-single-file-is-not-enough) but I think that should still probably be mentioned in the README as well. |
I opened the PR for this here: Qiskit/rustworkx#501 |
@mtreinish Thanks for the response. I will join the Slack space, but I would prefer to also have the option of using some asynchronous communication channel that's public and keeps a record of the conversation (for the sake of the review). Is there a place where I can ask my questions that way? Should I just use the issue tracker? Have you considered GitHub Discussions? It's just an idea.
The support channels should be communicated clearly to normal users, not just would-be contributors. The README is a much better place for this. The PR looks good! |
@szhorvat If I follow your comments correctly, I think you should feel free to use the issue tracker for your review comments. (I'm the rotating associate editor in chief for the week and trying to answer questions.) |
For reference, the paper is here on arXiv: https://arxiv.org/abs/2110.15221 |
Regarding the name The API is different (not a replacement), the projects appear unrelated, and retworkx doesn't appear to be closer to networkx than any of the other Python network libraries. I am wondering if you have contacted the networkx project regarding the naming, and if they are happy with it. I am not sure if this is within the scope of a JOSS review, but I thought it important to mention. |
I have now read the software paper, and I believe it needs a bit of improvement. In my view, the purpose of a JOSS paper is so that readers can get answers to questions such as:
As the paper currently stands, these questions are not fully answered. Statement of needHere, the authors state their own need, when working on the Qiskit project (networkx was used originally, but it was too slow, so retworkx was created to replace it). I would expect to read about the target audience's need. The broader target audience was set in the abstract: retworkx is "a flexible graph and network analysis library". I believe that the majority of people who are interested in such libraries will not have heard of Qiskit, thus talking about the needs of Qiskit will not be meaningful to them. In short: What so-far unmet need does retworkx aim to fulfil in the field of graph and network analysis? Related workIn this section, the authors list a few (but not all) alternative libraries, and simply state that they were not suitable for the needs of Qiskit. Some of the issues are:
I should note that the landscape of graph libraries is very crowded in Python. It might seem like it is hard to stand out. Yet, as a co-developer of igraph, it is quite clear to me that it is nearly impossible to create a truly all-purpose graph library. There will always be tradeoffs, and between the many library choices there are gaps to fill in. But which of the gaps does retworkx fit into? I cannot tell after reading the paper. To give some examples of what could be discussed, some libraries are targeted at graph theory researchers (mathematics). SageMath should be mentioned here. Some others (including most of those mentioned in this section) are mainly for complex network analysis, a very different field. In both these fields, users are likely to work with static graphs and use existing algorithms instead of implementing new ones. In contrast, some libraries may strive to provide a data structure for implementing efficient graph algorithms. There are many possible representation of graphs on a computer, all coming with their own tradeoffs: e.g. efficient modification is in conflict efficient graph traversal. Some libraries even provide multiple underlying representations in an attempt to deal with this. Please clarify these questions in this section. SageMath and NetworKit should be mentioned. I would like to bring the authors' attention to LEMON graph as well, which stands apart from most other libraries with its focus on optimization and enabling the implementation of efficient algorithms. LEMON is C++ but has a (now abandoned?) Python interface. The decision to write everything from scratch, as retworkx seems to do, is a major one. It will take a very long time to catch up with other libraries, functionality-wise. It would be good to discuss why this decision was made instead of building on top of an existing solid library such as LEMON. Graph data structuresThis is a good section because it aims to explain some of the conceptual foundations. It could be improved by adding some examples (e.g. to clarify what "callbacks" are). I suggest to take the opportunity to also explain the core capabilities of the library: libraries may support simple graphs, multigraphs with self-loops, directed/undirected or even mixed graphs, weighted graphs, vertex and edge attributes. Which of these features are or aren't supported? It would be nice to touch on the tradeoffs I mentioned in the previous section, such as tradeoffs between efficient modification and traversal. How is the graph represented internally? Use CasesThis section is in fact about benchmarks that compare with igraph, graph-tool and networkx. It would be good to also discuss some actual potential use cases, and mention examples of what tasks one may solve with the library. The benchmarks are hosted in a different repository under a different user: https://github.com/mtreinish/retworkx-comparison-benchmarks If the paper gets accepted, the current contents of the main repository will be archived along with the paper. With this organization, it looks like the benchmarks might be left out, which will be a problem for future reproducibility. This should be remedied. (@drvinceknight, correct me if I'm wrong about this.) In fact, some of the benchmarks are flawed (I will open issues for this later—but in which repository should I do this?). If a reader would only see the benchmark results, without an opportunity to look at the benchmark code as well, they will come away with an incorrect impression ... all the more reason to bundle the code. It's a bit unclear what is being compared between libraries. How fast they accomplish a task (with whatever algorithm they have implemented)? Or how fast the implementation of a specific algorithm is? Sometimes different algorithms are used (e.g. VF2 vs VF2++). The benchmark intention should be made clearer. The benchmark tasks should be described in more detail, e.g. with shortest paths: Weighted or unweighted? Directed or undirected? These do make a difference in some libraries. Single-source shortest path (implying all other vertices are targets) or the path between one source and one target (the search may be terminated as soon as the target is reached)? There are only three benchmark tasks. What do we learn about the libraries from these? Why were these specific tasks chosen, and do you expect the performance findings to generalize to other tasks? In the case of subgraph isomorphism, I would say no. For shortest paths, they might, as many typical calculations on graphs rely on the same kind of graph traversal as in shortest path finding. However, the shortest path benchmark might be backed up with some others that exercise the library in similar ways (e.g., betweenness). SNAP is omitted from the comparison "because its Python wrapper did not contain the required functions" (see footnote). Does this refer to VF2? If so, I do not think this is a fair reason to exclude it. Subgraph isomorphism is rarely needed for practical network analysis, and the fact that it's not implemented shouldn't immediately disqualify a library even from the other benchmarks. |
Graph Creation benchmarkThis benchmark is severely flawed. The With igraph, the correct way to create a large graph like this is to first build an edge list as a list of tuples of vertex indices, then call Note that building the graph this way may take less time than reading the entire huge gzipped file line by line using Python. Thus, it makes sense to separate the timings for reading the file vs building the graph. This brings us to the question: what does it even mean to time "creating a graph from an edge list"? The performance will depend on how that edge list was originally stored, and whether it first had to be converted to a suitable input format for a given library. |
@drvinceknight: This is my final report. In the past few days, I have been thinking about how to go forward with the review. My opinion is that the submission to JOSS was premature. This package certainly shows promise, and I think it would make sense to resubmit it in the future, maybe in a year's time. However, trying to work out all the issues within the timeframe of the review does not seem realistic. It would put too much pressure on the authors, as well as on the reviewers. Software paperPlease refer to my comments above. In short, as currently written, the paper does not provide enough information to readers to decide whether the library would be useful for them. I leave the checklist items for "Software Paper" unchecked for now (quality of writing and references would need to be re-checked anyway once the paper is updated). BenchmarksThe library appears to be fast, but there are two issues with the benchmarks, as presented:
More comments are above. I left the Performance item unchecked on the list for now. The libraryAccording to the submission, retworkx aims to be a new general-purpose Python library for working with graphs. In my comments about the paper, I discussed how different use cases could be targeted within this broader goal. Possibilities are:
It seems to me that at this time, the library is not practically usable for either of these three tasks. There are too many functionality gaps. For example, even decomposing to connected components is not implemented for undirected graphs (only for directed ones). I quickly found bugs in fundamental functionality, such as computing degrees (now fixed). The core API misses features critical for implementing graph algorithms (e.g. getting incident edges). Other examples are in the issues I opened. I could not check off the Functionality item at this point. The documentation is somewhat haphazardly written, with many typos and sometimes unintelligible descriptions. The same applies to the naming used in the API, e.g. I could not check off several of the Documentation items. All that said, the library is built on solid foundations (the Advice to the authorsWhile the landscape of Python graph libraries is crowded, there is definitely space for newcomers. However, new libraries should have a vision that distinguishes them. Which gap are you trying to fill? Think about what you are trying to achieve, and instead of letting the library grow organically (one might say haphazardly), think about a coherent design that fits the proposed use case. It seem to me that retworkx's biggest problem is that it has a single driver: being a component in Qiskit. If it does just that—serve very specific needs within a single project that does not even deal with networks—it is unlikely to become useful to a broader audience, and would not be publishable. I elaborated on this a bit in my comments about the software paper. Let actual general use cases be the driver behind development. Would you like it to be useful for analysing network data? Then you, the developers, should complete a number of network analysis projects. That will show you what is missing (e.g. can you do the first step, import data from common formats?) Talk to people who work in network science. Do you want it to be useful for prototyping graph algorithms? Then do that: implement algorithms not in Rust, but in pure Python using retworkx. Graph theory? Read Math.StackExchange or even MathOverflow, find computational tasks, see if you can solve them. Finally, you have the advantage of starting from scratch. You are not carrying all the heavy historical baggage of other graph libraries like igraph or networkx. First, understand the other libraries' limitations, understand the mistakes they made, learn from them. Come up with a better design. I see functionality being piled on, but very little design. Why are you foregoing this unique opportunity? To be frank, the software paper gave me the impression that retworkx was born out of a "not invented here" mindset, not from a detailed understanding of what is missing in existing Python graph libraries. |
First, I want to thank you for your detailed look at the retworkx so far. I was hoping the paper review would serve as a chance for people with different sets of experience and viewpoints to review the library and point out gaps and all the issues you've opened will help us make a better library. One of my personal goals with retworkx recently is to expand it's usefulness for more domains including a more traditional network analysis and getting your perspective on this has been quite valuable and you've identified a lot of places we can make improvements. But, I think because your critique here is coming from a completely different viewpoint you're inadvertently missing the use cases for which retworkx was developed and the value it's providing. What you're attributing to a lack of good design are just different priorities and design decisions we've made in the development of retworkx. This doesn't preclude making the changes you're looking for, but it also doesn't invalidate what has been done so far or how retworkx is used today. In the development of retworkx so far we've been more focused on applications that use graph or network analysis as a component or tool for a specific larger goal or application. This is why we've designed retworkx to be dynamic and flexible in its use. While retworkx was originally designed to meet Qiskit's needs from a graph library, Qiskit is not a single application or exclusively what retworkx can be used for. More specifically, Qiskit is a constellation of different software projects each working on a different aspect of quantum computing, including applications of quantum computing. The original use case for retworkx was in the internal use by the compiler where a directed acyclic graph is used to represent a quantum circuit (i.e. a program for a quantum computer). The compiler analyzes that graph and performs transformations on it to fit and optimize the input program to the target QPU. This is where we hit scaling limits, which prompted the development of retworkx. In this application dynamically building and modifying large graphs with thousands of nodes fast is very important as it's the bottleneck for compilation. Another application of graphs in Qiskit's compiler uses a directed graph to model the connectivity of qubits on a QPU , which is one of the hardware constraints the compiler needs to adapt an input program to. An example application of this connectivity graph we recently just developed was a new compiler technique applying subgraph isomorphism to attempt to find a perfect mapping of the qubits used in a quantum circuit to the physical qubits in a QPU. But this is just in the base layer of Qiskit (the interface between software and hardware) and there are higher level applications of quantum computing which are using retworkx in different manners. For example, recently the Qiskit Nature project, which is concerned with applications of quantum computing for natural sciences, started using retworkx recently to create lattice models to perform physics simulations. Looking outside of Qiskit there are additional applications currently using retworkx such as qtcodes, which is a downstream user of Qiskit and retworkx directly. qtcodes is using retworkx to research new techniques for quantum error correction (you can see some examples of this here: https://github.com/yaleqc/qtcodes/blob/master/tutorials/rep/2-fitters.ipynb but I'm not an expert in QEC and won't be able to provide an adequate explanation of how it works). Another research project using retworkx is atompack (who were a very early user of retworkx) where they use retworkx to create graph models of atomic structures. Both of these (which we cited in the paper) are using retworkx to generate graphs dynamically as part of a larger application. As for things you highlight like inconsistencies in the API, bugs in certain functions, or issues in the documentation, many of those are valid things to fix, and highlight why getting multiple sets of eyes on software is so useful, and why I value this review process. But, for some of the issues you've outlined I think this is another place that highlights where we have a difference in priorities. One of the things we value strongly in retworkx is backwards compatibility. While the project may have only existed for a couple of years we already have a large user base both via Qiskit but also outside of it and we can't simply change an API to make naming consistent or change documented behavior without potentially breaking people's applications. For example, one of the quirks you've highlighted, the difference between For the paper, part of why we didn't go into too much detail there is partially because of the length limits. Reading the expectations of the JOSS paper we were cautious to try and avoid explaining all the internal details of the library and saving that for the documentation. The other aspect was we were trying to avoid putting any project in a negative light. We felt it was important to showcase how retworkx's performance compares to other popular graph libraries for python because the we felt the performance characteristics of the library is one of the key reasons you would use retworkx. But, we weren't out to paint any particular library in a negative light as they're all useful for different applications and tried to avoid going into too deep an analysis or comparison because the purpose of the paper is not be a survey comparing in detail different graph libraries in Python. But, it is also difficult to explain all these nuances in just ~1000 words. We'll try to tweak the paper based on your suggestions and update the review here with an updated draft when it's ready. For the specific thing about the creation benchmark, you're right we shouldn't have used the networkx converter there with igraph. This was done mainly because creating a graph with igraph in the same manner as all the other libraries benchmarked was exceedingly slow (taking hours to create the graph). As we're not experts in igraph we referred to the documentation and there was no reference that we could find mentioning what you mentioned with the performance characteristics of the |
Let me sum up my recommendation, to avoid misunderstandings. There is no doubt that retworkx shows a lot of promise, but it does not look mature enough. When trying to use it for various tasks, its limits are reached very quickly and the functionality gaps become apparent. It was easy to find bugs, which again suggests that it has not been thoroughly tested through real-world usage not related to Qiskit and its dependents. I think resubmission at a more mature stage would make sense. At that time, it would help to (1) state the scope clearly (2) demonstrate use cases within the scope. As a comparison, another library I reviewed recently has several worked examples in its documentation which shows that it can be used to solve new tasks within its scope. The paper word limit should not be an issue since many things are better placed in the documentation anyway. All the issues I brought up are fixable, and I hope to see retworkx succeed and eventually published in JOSS. However, in my opinion, this would take much more time than what is reasonable during a review. I do not think that it is in the authors' interest to try to address all concerns under time pressure, especially since parts of the library already make the impression that they were created in a rush. Also consider that retworkx, like most graph libraries (and perhaps unlike most JOSS submissions), has a very large scope, thus it simply takes more time and effort to bring it to a reasonable level of completion. I will address some of the specifics in @mtreinish's response at a later time. |
@drvinceknight I don't seem to have the ability to update the checklist in the initial issue post. I have completed my initial review of the software and paper. General commentsI'm glad to see Rust being used as a safe and fast back-end for serious computation, and I have no doubt that this fulfills the needs of the Qiskit project. However, I too share some of the concerns of the other reviewer about the utility of the library for a more general audience, and to the benchmark methodology. A cursory look at the reference seems to suggest that retworkx has some support for other graph algorithms such as cutting and colouring, but there are not mentioned at all in your benchmarks, and the support looks relatively limited as compared to networkx, igraph, or graph-tool. To be clear, I would not expect feature parity with these very mature libraries and this is not a criticism that I put much weight behind, but I would perhaps like to see benchmarks for all of the functionality you have (and that are stable) against these other libraries to show some direct benefit of retworkx over other tools. The reason I mention this is that the name of the library "retworkx" implies a fast alternative to "networkx", but in reality only a small subset of the functionality is represented as of yet. DocumentationThe statement of need suggests a need for a drop-in replacement for the networkx library in which the core algorithms and data structures are passed to a compiled binary to do the actual work, thus improving the performance with minimal changes to the Python API. I agree that there is a place for such a library in the current ecosystem, and I am happy to accept that the external API of the other graph libraries mentioned do not fulfill the requirements for which retworkx was designed. I cannot comment on whether more general users of graph libraries in Python would find this to be true. The installation instructions provided in the documentation are excellent and, since this is distributed as a wheel containing the compiled binaries on my architecture and OS, I had no problems installing the library for testing. (This should be the case for most users, as wheels are available for Python 3.6-3.9 on most of the major architectures. However, since Python3.10 was release recently, the authors may wish to consider adding a wheel for Python 3.10 too.) The API documentation is fine, and the core classes have plenty of example usage, though I have not checked exhaustively. However, the algorithm functions have rather sparse documentation. It would be good to have at least one example usage in each of the main algorithm methods, even if this is a trivial example, so users can see what a typical call might look like. I would like to see more examples from a top level that walk through how to use the library to construct, manipulate, and use graph objects in a more complete pipeline; I have always found such "tutorial" style documentation to be a great way to cover a great deal of the usage without users needing to dig into API documentation. The library has a suite of tests, which as far as I can see test results against networkx (again, I have not checked every test file). This is fine, assuming that networkx does not contain any bugs. I would suggest, if there are not already, adding tests that explicitly construct correct results to compare computed results against rather than relying on networkx to test the correctness. You have documentation about how people might contributed to the library, but nothing to suggest how users should raise issues. I assume, since the project is hosted on GitHub, that they should create an issue on the repository. I would add a small section to the end of the Readme document to confirm this, or point users to the place that they can file a bug report. (This is set on the "bug tracker" attribute on PyPI.) FunctionalityAs I mentioned, I had no problems installing the library from wheel. However, I noticed that the setup.py file requires the rust-specific setuptools addon yet this is not listed as a build-time dependency. Rather, the setup.py file attempts to load this addon, and if it fails uses a subprocess call to pip to install the necessary package and then attempts to proceed. I understand why the authors have structured the code in this way, but I don't think this is a good idea. First, the pip-install step might fail and then users would get a cryptic error message when the setup resumes. Second, some users might be using tools to scan dependencies, and this would be missed. This is something the setup.py file format doesn't address well, and I can't offer a better solution. As for functionality, there are some small inconsistencies between the interfaces of retworkx and it's inspiration networkx that users might find a little frustrating (though nothing, as far as I can see, that would cause problems). At present, retworkx does not cover the same range of functionality as its competition, though that could hardly be expected (as mentioned above, those libraries are very mature). I am in agreement that the benchmarks on performance leave something to be desired, though I have not delved into them quite as deep as the other reviewer. I won't reiterate the comments about construction benchmarks, though these do need to be addressed. Instead, I will mention that the benchmarks should cover a wider range of use cases. The graphs used in benchmarking are generally pretty large. I would suggest that a large number of graph library users would primarily work with graphs that are relatively small by comparison (a few hundred or fewer nodes). Your benchmarks don't address this end of the spectrum at all. I would consider adding benchmarks for these cases too, although I expect the performance gap to be considerably closer here. Software paperGenerally I don't see any issues in the software paper. The summary and statement of need are clear. The code for the benchmarks should be included in the main repository, so it is more strongly linked to the version of retworkx. I suggest making a folder called "benchmarks" and placing the code in there. |
@IvanIsCoding in your paper.md file, you have 2 figures that use the same label ( |
I have updated the label, it should work now |
@editorialbot generate pdf |
@editorialbot recommend-accept |
|
|
👋 @openjournals/csism-eics, this paper is ready to be accepted and published. Check final proof 👉📄 Download article If the paper PDF and the deposit XML files look good in openjournals/joss-papers#3659, then you can now move forward with accepting the submission by compiling again with the command |
Sorry all, I now realize I may have jumped the gun slightly, as I'm not sure that the reviewers are fully satisfied with publication, based on the checklists and my review of the full issue here, so I will ask next before we proceed. |
👋 @szhorvat, @inakleinbottle - can you confirm that you are ready for this submission to be accepted and published? |
Confirmed. |
👋 @inakleinbottle - can you confirm that you are ready for this submission to be accepted and published? |
@danielskatz Sorry for the delay. Yes, all of my concerns have been addressed. I still cannot tick the boxes in the review post which is why they are not filled in. |
👋 @IvanIsCoding - I've now proofread the paper, and have made suggestions for small changes in Qiskit/rustworkx#720 - please merge this, or let me know what you disagree with, then we can finish the process |
@editorialbot recommend-accept final check before publishing... |
|
|
👋 @openjournals/csism-eics, this paper is ready to be accepted and published. Check final proof 👉📄 Download article If the paper PDF and the deposit XML files look good in openjournals/joss-papers#3661, then you can now move forward with accepting the submission by compiling again with the command |
@editorialbot accept |
|
🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦 |
🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨 Here's what you must now do:
Any issues? Notify your editorial technical team... |
Congratulations to @mtreinish and @IvanIsCoding (Ivan Carvalho) and co-authors!! And thanks to @szhorvat and @inakleinbottle for reviewing, and to @drvinceknight for editing! |
🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉 If you would like to include a link to your paper from your README use the following code snippets:
This is how it will look in your documentation: We need your help! The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:
|
Submitting author: @IvanIsCoding (Ivan Carvalho)
Repository: https://github.com/Qiskit/rustworkx
Branch with paper.md (empty if default branch): joss-paper
Version: 0.12.0
Editor: @drvinceknight
Reviewers: @szhorvat, @inakleinbottle
Archive: 10.5281/zenodo.7158473
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@szhorvat & @inakleinbottle, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @drvinceknight know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Review checklist for @szhorvat
✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
Review checklist for @inakleinbottle
✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
The text was updated successfully, but these errors were encountered: