Reapply preferred_dir fix, now with better backwards compatability

[vidartf]

This reverts #1140, reapplying #983. It then adds compatibility with jupyterlab_server prior to jupyterlab/jupyterlab_server#324, by writing back validated values to the serverapp's trait.

[codecov]

Codecov Report

Base: 79.93% // Head: 79.88% // Decreases project coverage by -0.04% ⚠️

Coverage data is based on head (7d14d76) compared to base (88c9339).
Patch coverage: 88.88% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1162      +/-   ##
==========================================
- Coverage   79.93%   79.88%   -0.05%     
==========================================
  Files          68       68              
  Lines        8099     8124      +25     
  Branches     1599     1601       +2     
==========================================
+ Hits         6474     6490      +16     
- Misses       1202     1207       +5     
- Partials      423      427       +4     

Impacted Files	Coverage Δ
jupyter_server/services/contents/manager.py	85.29% <84.21%> (-0.05%)	⬇️
jupyter_server/services/contents/filemanager.py	75.27% <91.30%> (+0.71%)	⬆️
jupyter_server/serverapp.py	80.01% <100.00%> (-0.16%)	⬇️
jupyter_server/services/contents/fileio.py	90.10% <100.00%> (+0.10%)	⬆️
...ter_server/services/kernels/connection/channels.py	58.37% <0.00%> (-1.33%)	⬇️
jupyter_server/services/kernels/kernelmanager.py	81.17% <0.00%> (-0.31%)	⬇️
jupyter_server/utils.py	85.29% <0.00%> (+0.58%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[blink1073]

The check_release failure is addressed in #1163

[blink1073]

Thank you!

[blink1073]

@meeseeksdev please backport to 1.x

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 1.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 8e4b7edcb7d812abb8497cf292cba34bc411bcf9

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #1162: Reapply preferred_dir fix, now with better backwards compatability'

4. Push to a named branch:

git push YOURFORK 1.x:auto-backport-of-pr-1162-on-1.x

5. Create a PR against branch 1.x, I would have named this PR:

"Backport PR #1162 on branch 1.x (Reapply preferred_dir fix, now with better backwards compatability)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.