X Tutup
The Wayback Machine - https://web.archive.org/web/20211103034629/https://github.com/PowerShell/PowerShell/issues/15552
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let's move PSModulePath out of the documents folder #15552

Open
JohnLudlow opened this issue Jun 9, 2021 · 18 comments
Open

Let's move PSModulePath out of the documents folder #15552

JohnLudlow opened this issue Jun 9, 2021 · 18 comments

Comments

Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
@JohnLudlow
Copy link

@JohnLudlow JohnLudlow commented Jun 9, 2021

Summary of the new feature/enhancement

The documents folder has always been where modules user-scoped are kept, and that's always been ok, but I recently discovered some problems with this as I've been getting more familiar with Azure and my work laptop maps Documents to OneDrive. This can be problematic for a few reasons, such as OneDrive not syncing correctly or trying to restore a bunch of files from a module I just uninstalled. This is particularly an issue with larger modules (or groups of modules) such as Az.*

We can certainly wish OneDrive was better at its job, but really these files shouldn't be there - they are not documents. The user is not expected to edit them or look at them. They are more akin to application files or application data. There's no value to them being in OneDrive other than to transfer them to another machine, and that can be better achieved by having a script in OneDrive that calls PSDepend or PowerShellGet to install modules on that other machine.

I have updated my profile script to remove this path but it reappears after running install-module.

Proposed technical implementation details (optional)

  1. Remove Documents\PowerShell\Modules from the $env:PSModulePath default. Select a new default path such as ~\PowerShell\Modules or $env:localappdata\PowerShell\Modules
  2. Offer a cmdlet or documentation to move the existing modules to the new location, and optionally update the value of the environment variable.
  3. Additionally, offer to move the existing modules and update the value of the environment variable during the install. (That way the user has the choice to make the install take care of it or do it later at a time of their choosing).
@jeroenlandheer
Copy link

@jeroenlandheer jeroenlandheer commented Jun 27, 2021

A few extra things to consider when putting this up for triage:

  • When I install a module on one machine and I am later at a different PC, OneDrive updates the modules folder... but Upgrade-Module doesn't work anymore, because it says the module wasn't installed with Install-Module ... (which it was, just not at the machine I'm working on.)
  • If your module folder is big loading PowerShell can take a (very) long time because it will wait until OneDrive has pulled all the files to your machine.

I think this change should happen... hopefully soon. I think storage for these modules should be in the %userpfile%\AppData\Local folder, this is the place for things that should not get synced with other machines.

PS: I create PowerShell modules... I have a lot of them... and they are all on my OneDrive now. 😒

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Jun 28, 2021

@jeroenlandheer Do you mean Update-Module? If so you couls discuss in PowerShellGet repository.

@JohnLudlow
Copy link
Author

@JohnLudlow JohnLudlow commented Jun 30, 2021

@iSazonov Would the PowerShellGet guys be able to change the default value of $env:PSModulePath and update the installer to optionally update it on upgrade?

Though there should probably be a second issue in PowerShellGet to respect the value, not overwrite it.

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Jun 30, 2021

@JohnLudlow Any changes related to PSModulePath are breaking changes. We need to keep backward compatibility.

@jeroenlandheer
Copy link

@jeroenlandheer jeroenlandheer commented Jun 30, 2021

@iSazonov Sorry, that was a typo. This should indeed go there too, but don't you think they will choose the location of the modules where they install them... based on whatever this teams says it should be? (AFAIK there's no way to customize this with settings, but that's something for another day.)

If paths for loading the modules is the primary function of PSModulePath, this issue can be solved quite easily:

  1. Designate a folder somewhere in %LocalAppData%\...
  2. Tell others that installing modules in this folder is best practice.
  3. Put both paths in PSModulePath (i.e. %LocalAppData%\.. and %userprofile%\Documents\...)
  4. Ensure the new location is added to new installations in the PSModulePath environment variable.

No need to move stuff around, new things get installed in %LocalAppData%\.. and old modules that are still in the user's documents folder keep working. Maybe this doesn't even require a code change in PowerShell, just some changes in the installer.

And now we're on this subject, in Windows installations, modules which are installed on the local machine are installed by default in the Program files\PowerShell\Modules folder. I think the correct place for this should be %AllUsersProfile%\... (often called C:\ProgramData)

AFAIK the functionality of PSModulePath itself doesn't change with all of this, so this is technically not a breaking change.

Hope this helps.

@JohnLudlow
Copy link
Author

@JohnLudlow JohnLudlow commented Jul 1, 2021

@jeroenlandheer Indeed, and if I do want to move the older modules to the new folder (I probably would, in my case), there are some options:

  • Offer to do that in the installer
  • Offer a tool (such as a cmdlet) to do it post-install
  • Offer documentation that says "you can copy everything from <old location> to <new location>"

Any of those would work for me, but some people may appreciate it being done for them.

And you're absolutely correct IMHO about the Windows Modules. The only modules that should be there are ones that are installed as part of PowerShell and shouldn't be removed.

@SeeminglyScience
Copy link
Contributor

@SeeminglyScience SeeminglyScience commented Jul 1, 2021

AFAIK the functionality of PSModulePath itself doesn't change with all of this, so this is technically not a breaking change.

While I agree that it would be hard to claim this is explicitly a breaking change of an API, it would still be disruptive. There has never been a supported way of determining the module paths of different scopes, e.g. no PSModuleInfo.AllUsersModuleRepository or similar. Any tools looking to interact with the module directories explicitly has been required to hard code the path, so any location change here will break these tools.

(Note that I'm not necessarily advocating against it, but it's something that needs to be considered)

@rjmholt
Copy link
Collaborator

@rjmholt rjmholt commented Aug 11, 2021

Related: #7082 (comment)

@kilasuit
Copy link
Contributor

@kilasuit kilasuit commented Aug 11, 2021

Previously discussed in #8069 too

@rjmholt
Copy link
Collaborator

@rjmholt rjmholt commented Aug 11, 2021

So we reviewed this on the @PowerShell/powershell-committee today, but also this is something I've personally done some thinking on in the past.

The mangling on particular locations on the PSModulePath is something we've seen users impacted by repeatedly. For example: PowerShell/vscode-powershell#2824 (comment).

Another important point is that the CurrentUser and AllUsers module paths are actually configurable in the powershell.config.json:

string allUsersModulePath = PowerShellConfig.Instance.GetModulePath(ConfigScope.AllUsers);
string personalModulePath = PowerShellConfig.Instance.GetModulePath(ConfigScope.CurrentUser);

So there is a concept of both the default CurrentUser module path and the actual one.

However, there's currently no way to communicate the actual one, so the default is the implicit contract between PowerShell's module system (Import-Module/Get-Module/command lookup) and PowerShellGet. I was hoping to improve that in #7082, but even without that the contact being implicit means there's room to get it wrong.

Today it seems that both PowerShell 7 and PowerShellGet agree that the default CurrentUser module path lies under the MyDocuments path, but evidence suggests that this wasn't always the case.

The big issue now is that OneDrive configurations override the MyDocuments folder and cause this unexpected migration.

But additionally, the actual CurrentUsers module path could be overridden with a configuration. And this is something that things like the PowerShell Azure Functions worker would want to do given the radically different environment in which it operates. Except that there's no way for PowerShellGet to pick this up.

In terms of changing the default PSModulePath, there have been a lot of proposals to fix PSModulePath in various ways. Changing the CurrentUser path away from MyDocuments is going to break PowerShellGet, since it depends on the same location being used. However, the MyDocuments path clearly presents problems, and both the proposed alternatives ($env:LOCALAPPDATA\PowerShell\Modules and $HOME\PowerShell\Modules) make sense.

So in terms of recommendations:

  • The Engine Working Group has also been asked to review this, to go into a bit more technical depth.
  • An API of some kind that gives both the default and actual module path components is probably a good direction to go in.
  • In terms of backward compatibility, tooling can look for this API and fall back to the traditional defaults.
  • The committee didn't decide on whether or not the default path should be changed. We want a chance for the Engine WG to discuss this, and also want to talk to the OneDrive team to understand things a bit better here.

@JohnLudlow
Copy link
Author

@JohnLudlow JohnLudlow commented Aug 13, 2021

@rjmholt That's a great summary. Thanks.

We'll see what the Engine Working Group say

@JamesWTruher
Copy link
Member

@JamesWTruher JamesWTruher commented Aug 23, 2021

With behavior of OneDrive the WG agrees that we need to come up with a proper solution, however the scenarios have a high level of complexity which is exacerbated by the need that other PS tools also need to be able to use whatever solution is made available.

The WG proposes that PS should provide an API which allows tools to query for the default location of modules. The tool chain also needs to be able to query PS so the values are consistent.

We propose that an RFC is needed to define the behaviors, not only for PowerShell proper but for tools like PowerShellGet.

We think at least the following requirements need to be addressed in the RFC:

  • where is the default location for PSModulePath (system and/or user)
  • how may this default location be modified
  • how is the API used by PS and the other tools
  • how are misconfigurations handled (non-existing paths, multiple paths, etc)

@ShadowXVII
Copy link

@ShadowXVII ShadowXVII commented Oct 6, 2021

+1. Nuget doesn't have this problem because it uses Profile Root\.nuget -- though perhaps long term the %LOCALAPPDATA% would make more sense for the user modules for PowerShell.

It's like syncing a .git repo in OneDrive! Unnecessary.

Whichever solution comes up, it should be configurable for devices, i.e. via InTune or PowerShell so we can make it consistent for all our developers.

@mikhey
Copy link

@mikhey mikhey commented Oct 18, 2021

Would a dot folder work here possibly? i.e. ~/.pwsh/ OR ~/.pwsh-modules I'm thinking of about Powershell running on other systems, Linux MacOS etc - if it could be consistent across platforms that would help "yugely" too. Well, I think at least.

I came across this as I'm looking to considerably cut down my documents folder when looking at OneDrive backup and I was hoping there was a way I could have PSGet use another folder. I'm still looking into what my options for this are - not sure if just changing PSModulePath env variable alone would do the trick.

@JohnLudlow
Copy link
Author

@JohnLudlow JohnLudlow commented Oct 18, 2021

@mikhey Changing PSModulePath works until you do Install-Module which breaks it again

A ~/.pswh folder (or whatever it gets called) would work, just as long as it's not somewhere OneDrive will try and mess with it

@rkeithhill
Copy link
Collaborator

@rkeithhill rkeithhill commented Oct 18, 2021

My preference is ~\.pwsh (or ~\.powershell) on Windows. The Microsoft guidance is to use ~\AppData\Local\Microsoft\PowerShell or some folder like that under ~\AppData\Local. My problem with that is AppData is hidden which doesn't make for a great user experience when you tell folks to edit their ~\AppData\Local\Microsoft\PowerShell\profile.ps1 script. If they try to navigate to that file via File Explorer they're likely to get tripped up by the "missing" AppData dir.

Also, I notice that many other tools on Windows are now using ~\.<tool> folders:

image

@mikhey
Copy link

@mikhey mikhey commented Oct 18, 2021

@JohnLudlow yeah, I was thinking of making it a symlink, but OneDrive doesn't like that for the backups. After I realized that I had much more space than I thought it did, I just let it go to OneDrive.

While it may have become a non-issue for me regarding OneDrive, I do feel it is something that should be addressed.

This has probably been mentioned before, but off the top of my head a few things to look at for an implementation could be:

  • Configurable use profile file to explicitly set primary user modules install path - for backward compatibility/support
  • A PowerShell environment variable or method that can be queried to locate user modules (cross module dependency)
  • Upon first run of PowerShell or installing/updating/removing a module, the user should be offered to move modules to the "Out of Documents folder" location to the new "preferred"/standard user profile location.

@rjmholt
Copy link
Collaborator

@rjmholt rjmholt commented Oct 18, 2021

Would a dot folder work here possibly?

As @rkeithhill said, those dot folders pollute $HOME, and even though they're not displayed, they still need to be processed by programs that examine the Home directory, creating a sort of tragedy-of-the-commons problem where more and more applications putting their configurations and data in a directory based in the Home directory adds up to slower filesystem performance for unrelated tasks.

PowerShell actually already conforms to the XDG base directory specification on *nix, which is a standard intended to solve this problem. PowerShell data files, including modules, are stored in $env:XDG_DATA_HOME/powershell, which defaults to $HOME/.local/share/powershell.

The issue then is that Windows doesn't really offer a clear answer here, because its philosophy is to separate users from configurers/application installers.

Worth noting is that the XDG standard is configurable with a sensible default, and I imagine we want the same, we just need to weigh configurability against complexity (mainly for anyone developing against our decision) and performance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
X Tutup