Autograder Skipped or Not Running

[lmraine]

Hi everyone,

We're having an issue with the github classroom autograder not being triggered for student assignments. Here is the general order of what is happening:

1. Assignment and tests are created
2. Student accepts the assignment and the forked repo is created
3. Student clones the repo, pulls, and then edits in VS code
4. After pushing to their fork, the autograder does not run at all
5. I go into "Edit" the assignment, make no edits, and click update assignment
6. Student fetches/ pulls again. Then re-pushes code
7. The autograder runs the test and assigns points.

I'm not sure why the autograder not running the first time a student submits their assignment or why I have to "update" the assignment to force it to run. Am I missing something in the set up with the assignment?

[joaquinvanschoren]

Hi all, I'm having the same issue. Committing doesn't start the autograder at all. If I edit the assignment, make no edits, and click update assignment, the autograder workflow is pushed to the fork, but it doesn't run. If I check back later, it actually ran, but the test is skipped. If I try to rerun the job it skips again. Is this a temporary issue or am I doing something wrong?

[lmraine]

We actually were able to identify what might be the cause of this issue today. It looks like the .github folder is not being created in all students' repos. It is created is some but not all, so when they push, there's nothing actually there to run the tests. When you "edit" the assignment (with no edits) that new commit somehow triggers the .github folder to be created. However, this does not happen consistently so some students still will not have the .github folder in their repo.

[mohamedtalhaouii]

---

[samus-aran]

Hi @lmraine 👋

Thank you for bringing this to our attention. We've passed this along to our internal team to investigate and when we have any updates we'll update this thread. 👍

[lmraine]

Thanks @samus-aran!

I'm currently manually adding the .github folder with the autograding json and classroom yml files to the upstream fork so students can submit their work for this week, but I can replicate this new assignments.

[samus-aran]

Hi @lmraine & everyone 👋

We are happy to say that there has been a fix deployed which includes amending all the existing assignments. You should not be required to do anything.

If anyone else is still experiencing this please reply 👍

[lmraine]

Hey @samus-aran

As of yesterday, the .github folder was still not being created when students initially accept the assignment. I'm creating the assignment from a private repo which has several files including test files. Their forks initially contain all the appropriate files, but not the .github folder. I then have to "Edit" the assignment, make no changes, click update assignment, and the student refreshes their browser and the .github folder is created. I have to do this several times throughout the day, pretty much whenever anyone new accepts the assignment.

[hhu]

Yes, this is happening to my students as well. They don't have a .github folder when they first accept the assignment, but we can get it to appear if I update the assignment with no changes after they accept the assignment.

[oliverosz]

This workaround works for creating the autograding workflows. However, this way the workflow is not run automatically, as the commit by github-classroom bot doesn't trigger it per the rule in the workflow. So either the students or I have to manually dispatch the workflow in each repository, or it will run when the student pushes commits to it.

There is another problem: the scores are not shown on the assignment page next to the student names. I did the manual dispatch for each but the score is displayed only for 1 student out of 15.

[kc8se]

This is still a problem. Not every student repo has a .github directory. I, personally, experienced it today, but have seen it in a lot of the student repos.

I also experience the other problem that @oliverosz . It only detects the executed autograding sometimes, for some students, or perhaps it just happens very seldom.

These two problems make it impossible to use the autograding feature as intended. Which is sad, as I think it's nice. Could you or someone please take a look @samus-aran ?

[booleanchile]

Also, The "Enable feedback pull requests" option is not working anymore.

[locbaoodo]

I am still having the problem today with my class. They can edit the file but the autograder doesn't run at all. I tried it myself and didn't get it to work either. Clicking on Create A Pull Request will create a pull request to my template git repo instead of their private repo. This is very confusing

[pinventado]

We still have this issue. Some student repos have the .github folder, while others do not. Please advise.

[samus-aran]

Hey everyone! 👋

We are happy to share that a patch has been deployed addressing the bug affecting .github folders and essential files (autograding and classroom.yml) not automatically pulling to the student assignment forks upon creation (accepting the assignment).

This patch does not update already existing forks, teachers may require to edit or update those to include autograding.

All newly created forks since the patch should have autograding included and run as expected.

Also, @samus-aran is there a way to stop workflows from auto-disabling when the students fork the starter repo?

☝️ We are still working on this feature. We will post in this thread when we have an update 👍

[samus-aran]

Thank you @lmraine for your commitment sharing your experiences! We've passed this onto the internal team to investigate.

@mcknijak , @pinventado, @joaquinvanschoren, @locbaoodo Are you also experiencing some discrepancies since the latest update (February 29th)?

[aaleta]

Hi @samus-aran . I'm still experiencing this issue. A month ago I created an assignment and throughout this month the .github folder was not created automatically in about 10% of the students' repos.

Yesterday I created three new assignments for an exam we will do today and joined myself to test if they were working correctly. The .github folder was not created in my repo when I joined the first one. I followed @lmraine workaround (edit assignment, make no edits, save the assignment) and got the .github folder. Then, I joined the second and third assignments but in those cases the folder was properly created when I joined.

[samus-aran]

@aaleta & @lmraine are you able to share with us the names of your classrooms and the assignments that are still problematic? Alternatively you can open a GitHub Education support ticket (select GitHub Classroom as your option) with this info and share the ref here so I can highlight it to the team 👍

[aaleta]

So yesterday three groups of students took the assignments, and I kept track of the issues. The assignments are private so let me know if you need more information to access them, or if it's better to submit a support ticket.

The first assignment was:

https://classroom.github.com/classrooms/149118562-computacional-unizar/assignments/prueba-1a-el-oscilador

A total of 25 students joined between 10:00-10:10 (CET) (+me, 26). The .github folder was not created automatically by the bot in 12 of them. As I mentioned, I also didn't get the .github folder when I joined myself.

The second was:

https://classroom.github.com/classrooms/149118562-computacional-unizar/assignments/prueba-1b-el-oscilador

29 students joined between 15:05-15:15 (CET) (+me, 30). 7 of them did not get the .github folder.

The third one was:

https://classroom.github.com/classrooms/149118562-computacional-unizar/assignments/prueba-1c-el-oscilador

19 students joined between 17:15-17:25(CET) (+me, 20). 9 of them did not get the .github folder.

In all three cases, I opened the edit button, saved the assignment without making any changes, and the .github folder was automatically added to all the repos that didn't have it. The only issue is that then some students had to do a pull to sync their local repos before pushing their commits, but other than that there were no further issues.

[samus-aran]

Thanks @aaleta I've shared this with the team. 👍

[samus-aran]

Hi everyone! 👋

Thank you all for your feedback! We are happy to announce that we have deployed the feature that all newly created student repo forks will have workflows enabled by default requested by @mcknijak and other community members 🎉 cc/ @oliverosz

We still have users experiencing intermittent issues with their .github folders not being transferred across to their students forks which the team are still investigating.

But if you aren't affected by the lack of .github folder issue in student forks please let us know if you encounter any issues with the workflow being enabled by default update 👍

[oliverosz]

Now the workflows work properly if I supply the .github folder in the teacher repository. It gets copied into the students' forks and workflows are enabled. 👏 However, the "add deadline" commit was not made by classroom-bot in some of the repos.

I actually prefer this method to the online wizard for defining tests, as it's easier to copy test commands in the autograding.json than filling out separate forms for each test. The latter method returned an internal server error (500) when I tried to save the new assignment, and the tests were lost.

[samus-aran]

Thanks for your feedback @oliverosz 🙇

However, the "add deadline" commit was not made by classroom-bot in some of the repos.

But it was expected? I'll pass this over to the internal team to review. 👍

The latter method returned an internal server error (500) when I tried to save the new assignment, and the tests were lost.

Are you able to provide steps to reproduce this? What actions did you take?

[oliverosz]

However, the "add deadline" commit was not made by classroom-bot in some of the repos.

But it was expected? I'll pass this over to the internal team to review. 👍

I think it's the same problem as the .github folder not being created, as it is normally created by classroom-bot in another commit when autograding tests are added by the wizard, not in the .github folder of the source repository.

The latter method returned an internal server error (500) when I tried to save the new assignment, and the tests were lost.

Are you able to provide steps to reproduce this? What actions did you take?

I played around with it to see when it occurs. I only get an error if I add a test case with test name 1 at the Grading and feedback page. Weird. 😃 I left all other settings on default, tried with other settings and different test types as well, it's consistent: the test case name cannot be 1. If editing an already existing assignment, there is no problem with name "1" and it gets saved.

[lmraine]

Hey @samus-aran

We're still experiencing the issue with the .github folder being created when students initially accept an assignment. It is working for some students but not for others.

[samus-aran]

Hi @lmraine 👋

@aaleta & @lmraine are you able to share with us the names of your classrooms and the assignments that are still problematic? Alternatively you can open a GitHub Education support ticket (select GitHub Classroom as your option) with this info and share the ref here so I can highlight it to the team 👍

[breed]

i finally got tests to run (not get skipped) by removing the following line from .github/workflows/classroom.yml:

if: github.actor != 'github-classroom[bot]'

it seems a legit line, but it prevents tests from being run when tests are updated.

[seemong]

Sadly I tried your fix, but it did not work for me.

[JRDollar31]

Hi @samus-aran,

Could you please have a look at this post? I am sorry if I put it to a wrong section. Any help would be greatly appreciated!

[samus-aran]

Hi everyone! 👋

Just to provide an update (I'll be marking this as an answer to make it more visible). The internal engineering team are still investigating the problem which we believe is an internal race condition. When we have an update we'll post here and re-mark the answer.

Thank you all for your patience and continuing cooperation. We still would love for you to comment and share with the team the classname, and assignments that are experiencing this issue so we can review.

[samus-aran]

Hi everyone again 🌞

We are happy to share that the engineering team have deployed an additional patch to address the bug affecting .github folders and essential files not being copied over as expected. Shipped Wednesday 13th March 🚢

This patch does not update already existing forks, teachers may require to edit or update those to include autograding.

All newly created forks since the patch should have autograding included and run as expected.

Please reply to this thread if you are still experiencing issues with missing .github folders when new student assignments are accepted (and creating new repos). 👍

cc/ @lmraine @breed @aaleta @oliverosz

Thank you all for your patience and continuous feedback to help us identify a path forward.

[ashishkeshan]

@mponce1uoc what was the error you were getting when you had the number first?

[mponce1uoc]

@ashishkeshan , yes the issue was only with test with the number first.

[ashishkeshan]

@mponce1uoc we pushed a fix for that. We now do not allow numbers at the beginning of the test name.

[gcarvalh-irsc-edu]

The autograder still isn't working for me. Multiple student repositories do not have the .github folder when the assignment is accepted.

For example:
https://github.com/GC-IRSC/project-1-5-practice-debugging-georgiana-carvalho
https://github.com/GC-IRSC/project-2-1-practice-stream-manipulators-georgiana-carvalho

Any updates on this issue? Or any workarounds?

[tturin1017]

Hi, I am experiencing the same issue. Some students do not have the .github folder. Even when I manually add it and re-run the auto grading, it is skipped.

[luis-damas-isel]

At this time I do not have a clear vision on this issue status, because some very simple test configurations keep faling.
The changes recently done on how the autograder runs hides thes setup process result.
I have a very simple test that have a setup done with: make se01-test. After that, the test is run passing 1 as argument: ./se01-test 1.

I download the repo and all comands run localy without problems, making the executable and running it.

But the autograder fails because it says that the executable se01-test does not exist.

So, I assume that the make command did not run or fail, but I cannot see that log.

Changing the test... I remove the setup part and added to the run command.

Sometimes it runs both commands (make and the app) because we can see all the compilation commands and then the generated output. When both execute, the output is dirty by the output generated by the make utility (yes, I know I can use > /dev/null).

On other situations, no info can be found.

In such scenario, how can we check what setup is being done (if any)? And what is making the final executable not to run?
Thanks.

[luis-damas-isel]

This in an internal problem of github because I cannot duplicate the test.

[ashishkeshan]

@luis-damas-isel thanks for bringing this to our attention. Looks like we have a bug with our input parsing logic that needs to be fixed.

[ashishkeshan]

@luis-damas-isel we just merged in a fix for this issue, could you try again?

[LuisDamas]

Hi.
Now is working fine.
The only thing not working is that I cannot "Reuse assignment" witch always fails.

[chrissphinx]

Going to echo @LuisDamas report here, the "Reuse assignment" functionality appears to fail silently. I do not see any error given and the assignment is simply not created in the target classroom

[joaquinvanschoren]

Hi all, since a week or so, all the autograded test started failing - they just drop without reporting any errors. Below are some screenshots. The only thing I get is a warning about running pip. I use pip to install requirements in a requirement.txt file, but this has never given me issues before.

Any idea how to fix this?

[joaquinvanschoren]

@ashishkeshan Thanks! Could you please also check what's wrong here?
https://github.com/ML-course/assignment-3-joaquinvanschoren-3

All I get is 'FAILED submission_test.py::test_submission - Failed: Error while running notebook.'
but when I run it locally in a clean venv it runs fine.

[ashishkeshan]

Looks like there's a bug with our autograding-python-grader. We will roll back the code. In the mean time, could you change this:

 - name: submission_test
      id: submission_test
      uses: education/autograding-python-grader@v1
      with:
        timeout: 60

to be:

- name: submission_test
  id: submission_test
  uses: education/autograding-command-grader@v1
  with:
    test-name: submission_test
    setup-command: sudo -H pip3 install --upgrade pip; sudo -H pip3 install setuptools; sudo -H pip3 install -U -r requirements.txt
    command: pytest
    timeout: 60

[ashishkeshan]

@joaquinvanschoren did that fix your issue?

[joaquinvanschoren]

@ashishkeshan Yes, thanks! Can this fix be deployed to all other repositories for this assignment?

[ashishkeshan]

If you edit the assignment test to be a run_command test with those parameters in the UI, it should update the tests in every student repo.

[nineil-weber]

Hello everyone, I am having problems with the autograder. It seems it is not working correctly. A few weeks ago it was up and running correctly. Now I tried uploading the same solution on a different account and it doesn't work properly.
Link : https://classroom.github.com/a/lI9KlWLK

Thank u.

[ashishkeshan]

What's the error you're getting?

[ashishkeshan]

Hello everyone, we have fixed the issue with the output not showing for tests. Please let me know if you still run into any issues with it.

[ashishkeshan]

@nineil-weber will take a look at our parsing logic to see why that character is getting added. Thanks for bringing it to our attention.

[ashishkeshan]

@nineil-weber we just merged in a fix for this issue, could you try again?

[nineil-weber]

HI @ashishkeshan, The same problem still continues.

[ashishkeshan]

You'll need to recreate the assignment for the changes to take place

[nineil-weber]

hI @ashishkeshan , I just created the assignment again and everything is working correctly. Thank you so much.

[gpincheiraa]

Hi!
More repositories without .github folder when the assigment is accepted:

https://github.com/booleancl/m05c01-terraform-cristobaldominguez/commits/main/
https://github.com/booleancl/m05c02-terraform-cristobaldominguez/commits/main/

in this cases my workaround was go to assignment edit screen, press "update assignment" and then the folder is created.

[chrissphinx]

This has not been fixed AFAICT:

All my students are now seeing this error if they have started an assignment since the change pushed on March 13th. The issue is that the classroom folder inside of .github folder is not created by the bot. This is what a broken, freshly created repo looks like:

and this is what a repo looks like with properly functioning autograder tests, note that this folder was created two weeks ago on March 11th specifically:

@samus-aran can classroom team please revert this broken change?? Otherwise it appears that the workaround will have to be manually adding the classroom folder to each of the repos created after this date...

[chrissphinx]

Sorry, I spoke too soon here, it looks like entire workflow process is being reworked as the new .github/workflows/classroom.yml file appears to have the content of the .github/classroom/autograder.json file. Notably absent from steps in the workflow is - uses: education/autograding@v1 so it seems like we're trying to phase out using that and the two separate files approach? The workaround will have to be copying both workflows and classroom folders from known working repos until this is sorted I guess... ☹️

[chrissphinx]

Just want to finally confirm that moving folders from repos with known working autograder workflows does get around this issue.

[chrissphinx]

Also now discovering that making an assignment inactive (since it for some reason invokes the bot on everyone's repo) screws up the autograder for every repo under that assignment. So thanks a lot for that... 😞

[chrissphinx]

Going to just update my specific thread to note that a student has discovered that test cases that are named starting with a number appear to cause this specific failure. I am not going to modify test cases now as we are at end of semester, but will try to confirm this later on this month...

[kc8se]

Apart from the above issues, which still occurrs. The autograder also swallows all info now.

The test result information from pytest in our case, information which we could see, is now never displayed by the autograder. It completely makes Github Classrooms unusable to us as the test information gave valuable hints to the students about what to fix. Can you please show this info again? Perhaps in its own folded section if you want it to be a "second class citizen"?

[ashishkeshan]

The test results from pytest should be showing up, could you link me to the assignment you've created so I can check it out to debug with?

[bharlowRHC]

I had originally setup my Python unit tests using the run command and setup individual test cases to run via pytest with different point values per test. This would give students a way to gauge how successful they are with each test case and award partial credit.

The new pytest runner seems to have taken over all my run commands and is now forcing all test cases to run every time even if I had originally specified a single test case to run.

Is my only solution to really setup brand new test cases using the run_command using the same exact command as I originally had just to get my original setup back? In one class alone, I will have to update 126 test cases back to the run_command, so this is not appreciated mid semester.

[chrissphinx]

Doubtful you will receive a reply here, but classroom team screwing with autograder gh actions generation mid-semester is definitely a big mistake. Kind of baffling considering they took the right steps when it came to switching repos over to forking model instead of template repos (it was opt in this semester and won't go live until June 17th).

Just putting this here FYI: it seems you must not name test cases starting with a number as one of my students has discovered. This is what is causing mine to fail. It seems like you have a different issue and your cases are not named this way. Sorry you have 126 test cases to update :(

Perhaps you can script your browser to do this work? They have not exposed any API for dealing with test cases and the given UI is very clunky still...

[mponce1uoc]

Helo @bharlowRHC .
I got the same issue.
Here are the results from the Autograding:

To solve this issue, I need to switch from using the "Python" command to the "Run_Command". The Python command version is not functioning properly.

The new version utilizing the 'Run_Command' is indeed functioning properly

Even though this interim solution is effective, it's frustrating because it would be better to proactively inform GitHub Classroom teachers about these changes. This way, we can update all the repositories beforehand and avoid relying on our students to report these issues. It's important for us to stay informed about significant changes like this.

[lmraine]

Hey @samus-aran

We've had some exams and spring break so I haven't had any new labs. However, I assigned one today, and it showed the student had 1 successful check but all the tests failed. When you click on the details, it shows 0/0 but the repo shows the student the green check mark and a successful check.

[LuisDamas]

Done! :-)

[ashishkeshan]

@luis-damas-isel which repo should I look at?

[LuisDamas]

Classrooms | estsetubal-bd-2023-24-classroom-ctesp-amadora | 02-select-groupy-by-Template
user with problematic ratings ...658@.....

[ashishkeshan]

@luis-damas-isel I just checked it out and looks like clasroom is pulling the info from an older commit that received a score of 19/20. I think it didn't update with the later commits because it thinks they were submitted late. Did you have a deadline for the assignment?

[LuisDamas]

The original commit that was graded with 20/20 in the details and appear as 19/20 in the assignment details.
Than I forced a change in a file to force to re-run the autograder to see if the correct value appear (which is marked as "Late") but it remain with 19/20, also if you download the scores.

[georgiana-carvalho]

@samus-aran and @ashishkeshan
The autograder still isn't working. Multiple student repositories do not have the .github folder when the assignment is accepted.
For example:
https://github.com/GC-IRSC/project-1-5-practice-debugging-georgiana-carvalho
https://github.com/GC-IRSC/project-2-1-practice-stream-manipulators-georgiana-carvalho

Any updates on this issue? Or any workarounds? I tried to Update Assignment as suggested in previous threads but that did not work for me.

[NuttKatyISD]

I've got to get some labs up for next week. What did seem to work was to include the .github/workflows/classroom.yml file in the source repository. From the students' side it doesn't look any different.

Only thing is that it doesn't trigger the feedback PR to create.

Not ideal, but hopefully will at least them to work next week.

[georgiana-carvalho]

That worked. Thanks for your suggestion.

[nineil-pitt]

Hi, the autograder of my classroom is not working.... i create another assignment, more simple and still not working.

My assignment : https://classroom.github.com/a/kV_RU2Lz
simple assignment : https://classroom.github.com/a/n6jGRMZJ

Some days ago, i tested and it was running ok.

[JoshIsomLehigh]

Not working for me either.

[carloscares]

It doesn't seem to be working. I have a weekly assignment with the exact same configuration. The autograding worked fine on August 12, but it isn't working on Tuesday, the 19th. Tomorrow, I plan to do a manual review :-(. My only hypothesis is that it's a temporary issue, and the only temporal cause I can think of (aside from bugs introduced in updates) is that the available free autograding minutes have run out. However, I can't find any information to confirm if this is the problem

[ericmercer]

I am seeing the same issue. The auto-grader is not working for the new classroom and assignments that I have created. It did everything as expected 2 weeks ago, but now, when students accept the assignment, the .github/workflow/classroom.yaml is not being created in the new repositories. Editing the assignment, and the auto-grading tests, does not change that (nothing to sync and new repositories still do not have the work flow).

The semester starts next week. It would be nice to have this sorted before then!

[ericmercer]

@samus-aran, are you able to look into this again?

[ericmercer]

Here is an example where the student that accepted the assignment two weeks ago has the workflow for the auto-grader while the one that recently accepted it does not: https://classroom.github.com/classrooms/178334895-byu-cs-236-preview/assignments/project-0-v3. Nothing has changed.

[ericmercer]

The Python auto-grader is working, again. But it is concerning that things quietly break and fix without much (any) communication to instructors.

[hlucia79]

Nothing seems to be working, again! Students are now getting a 500 Internal Server Error after accepting an assignment.
Any suggestions?
I had assignments that were working just fine last week. I did not change anything and now none of the assignments can be accessed by the students. As I said, after the students are accepting an assignment, they are getting a "500 Internal Server Error. We're sorry, but something went wrong. Try going back to classroom dashboard"

[chrissphinx]

@hlucia79 check this thread: https://github.com/orgs/community/discussions/130738

someone is pushing untested code to production again apparently 🤷

[AncestorComposition]

There is a discussion on this bug at https://github.com/orgs/community/discussions/137628

[szschaler]

This is still not working fully. I have just set up a Github Classroom assignment. This doesn't use autograding, but the template repo does contain an action to be run when a repository is created. The workflow file is correctly copied into the student's repository, but Github doesn't recognise that there is a workflow there at all. When I go to the Actions tab, no workflows are shown at all. Students have to manually open the workflow's YML file, make an arbitrary edit, undo the edit, and commit the file to get Github to recognise the action. Then they have to manually run it, because it now wasn't present when the repository was created (as originally designed)...

What are the plans for fixing this issue? It's truly messing up Github Classroom.