Native invocation using `ArgumentList` #14692

JamesWTruher · 2021-02-02T01:23:03Z

This adds a new behavior for native executables which uses ArgumentList as the command arguments for the native executable.

PR Summary

The experimental feature PSNativeCommandArgumentPassing when enabled will use the ArgumentList property of the StartProcessInfo object rather than our current mechanism of reconstructing a string when invoking a native executable.
$PSNativeCommandArgumentPassing is a new automatic variable which allows for runtime selection of behavior. $PSNativeCommandArgumentPassing may be set to either Legacy or Standard, Legacy is the historic behavior. The default when the experimental feature is enabled is the new Standard behavior.

This new behavior is a breaking change from current behavior. This may break scripts and automation which work-around the various issues when invoking native applications. Historically, quotes must be escaped and it is not possible to provide empty arguments to a native application. This PR attempts to address these issues.

new behaviors made available by this change

literal or expandable strings with embedded quotes the quotes are now preserved

PS > $a = 'a" "b'
PS > $PSNativeCommandArgumentPassing = "Legacy"
PS > testexe -echoargs $a 'a" "b' a" "b    
Arg 0 is <a b>
Arg 1 is <a b>
Arg 2 is <a b>
PS > $PSNativeCommandArgumentPassing = "Standard"
PS > testexe -echoargs $a 'a" "b' a" "b    
Arg 0 is <a" "b>
Arg 1 is <a" "b>
Arg 2 is <a b>

empty strings as arguments are now preserved:

PS>  $PSNativeCommandArgumentPassing = "Legacy"
PS> testexe -echoargs '' a b ''           
Arg 0 is <a>
Arg 1 is <b>
PS> $PSNativeCommandArgumentPassing = "Standard"
PS> testexe -echoargs '' a b ''           
Arg 0 is <>
Arg 1 is <a>
Arg 2 is <b>
Arg 3 is <>

the new behavior does not change invocations which look like this:

PS> $PSNativeCommandArgumentPassing = "Legacy"                                            
PS> testexe -echoargs -k com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect          
Arg 0 is <-k>
Arg 1 is <com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect>
PS> $PSNativeCommandArgumentPassing = "Standard"                                  
PS> testexe -echoargs -k com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect
Arg 0 is <-k>
Arg 1 is <com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect>

Additionally, parameter tracing is now provided so Trace-Command will provide useful information for debugging

PS> $PSNativeCommandArgumentPassing = "Legacy"                                           
PS> trace-command -PSHOST -Name ParameterBinding { testexe -echoargs $a 'a" "b' a" "b }
DEBUG: 2021-02-01 17:19:53.6438 ParameterBinding Information: 0 : BIND NAMED native application line args [/Users/james/src/github/forks/jameswtruher/PowerShell-1/test/tools/TestExe/bin/testexe]
DEBUG: 2021-02-01 17:19:53.6440 ParameterBinding Information: 0 :     BIND argument [-echoargs a" "b a" "b "a b"]
DEBUG: 2021-02-01 17:19:53.6522 ParameterBinding Information: 0 : CALLING BeginProcessing
Arg 0 is <a b>
Arg 1 is <a b>
Arg 2 is <a b>
PS> $PSNativeCommandArgumentPassing = "Standard"                                            
PS> trace-command -PSHOST -Name ParameterBinding { testexe -echoargs $a 'a" "b' a" "b }
DEBUG: 2021-02-01 17:20:01.9829 ParameterBinding Information: 0 : BIND NAMED native application line args [/Users/james/src/github/forks/jameswtruher/PowerShell-1/test/tools/TestExe/bin/testexe]
DEBUG: 2021-02-01 17:20:01.9829 ParameterBinding Information: 0 :     BIND cmd line arg [-echoargs] to position [0]
DEBUG: 2021-02-01 17:20:01.9830 ParameterBinding Information: 0 :     BIND cmd line arg [a" "b] to position [1]
DEBUG: 2021-02-01 17:20:01.9830 ParameterBinding Information: 0 :     BIND cmd line arg [a" "b] to position [2]
DEBUG: 2021-02-01 17:20:01.9831 ParameterBinding Information: 0 :     BIND cmd line arg [a b] to position [3]
DEBUG: 2021-02-01 17:20:01.9908 ParameterBinding Information: 0 : CALLING BeginProcessing
Arg 0 is <a" "b>
Arg 1 is <a" "b>
Arg 2 is <a b>

PR Context

Invoking native applications is sometimes very problematic because of how the application is invoked. A new property is available on the ProcessStartInfo object which hopes to improve the experience. This PR takes advantage of this new property.

PR Checklist

The impact on completions may be affected if those completers are calling native applications.

iSazonov · 2021-02-02T18:34:49Z

/cc @mklement0 Please look the PR.

mklement0 · 2021-02-03T20:55:38Z

While switching to .ArgumentList is definitely called for and fully solves the problem on Unix-like platforms, on Windows there are also vital accommodations that we need to make for common use cases, notably calling batch files and misexec-style CLIs, as detailed in ~~this comment~~ [update: see Implement accommodations for Windows CLIs, notably batch files and misexec-style programs as part of experimental feature PSNativeCommandArgumentPassing #15143].
- If there's disagreement about these accommodations, let's have a discussion, but none has happened yet (except with other community members), so, yes, perhaps an RFC is necessary.
Both improvements should be part of the same PR, to once and for all solve 100% of all argument-passing problems on Unix, and 99% of all problems on Windows (the remaining problems cannot be fixed generically, and must be addressed by users via --%).
That is, only one (opt-in) breaking change should be made, and it should be controlled by a single preference variable (which is the next best solution if the breaking change is deemed unacceptable).
As for the preference variable:
- First, the name $PSNativeApplicationUsesArgumentList is too technical to begin with (end users who expect argument-passing in their shell to "just work" cannot be expected not know about plumbing details such as .Arguments vs. .ArgumentList, which would make discovery difficult); also, the variable should be [bool]-typed (while assigning 1 works through implicit conversion, we should show $true / $false in the docs).
- Second, with the PR encompassing the Windows-only accommodations too, a broader name is called for. I'm struggling to come up with a good suggestion. $PSFixNativeArgumentPassing?

SteveL-MSFT · 2021-02-03T23:38:46Z

For the variable, I prefer $PSNativeCommandArgumentParsing as an enum with values ArgumentList, and String (can't think of something better...).

mklement0 · 2021-02-04T15:25:23Z

@SteveL-MSFT:

I prefer $PSNativeCommandArgumentParsing

Not sure if it's just a typo, but to be clear: this isn't about argument parsing, it's about argument passing (which, on Windows only, also involves encoding, as a single string - which .ArgumentList will do for us, but the suggested accommodations will have to be coded manually).

an enum with values ArgumentList, and String (can't think of something better...).

I can see why you would want to avoid the word "fix" in the variable name - even though that's what it comes down to.

Again, what matters to the user is only the following - not technical characterizations based on plumbing users may not even know of:

Is the old behavior in effect, where I need to apply workarounds?
Or is the new behavior in effect, where things work as expected?

Thus, if you do want an enum, I suggest a $PSNativeCommandArgumentPassing variable with the following enumeration values, corresponding to the above:

Legacy
Standard (or similar)

SteveL-MSFT · 2021-03-03T23:41:17Z

@PowerShell/powershell-committee reviewed this, we agreed on automatic variable $PSNativeCommandArgumentPassing as an enum with values Legacy and Standard (default). This feature needs to be an ExperimentalFeature.

src/System.Management.Automation/engine/NativeCommandParameterBinder.cs

test/powershell/Language/Scripting/NativeExecution/NativeCommandArguments.Tests.ps1

src/System.Management.Automation/engine/InitialSessionState.cs

src/System.Management.Automation/engine/ExperimentalFeature/ExperimentalFeature.cs

src/System.Management.Automation/engine/InitialSessionState.cs

test/powershell/Host/ConsoleHost.Tests.ps1

src/System.Management.Automation/engine/InitialSessionState.cs

test/powershell/Language/Scripting/NativeExecution/NativeCommandArguments.Tests.ps1

SteveL-MSFT

LGTM

TravisEz13 · 2021-03-24T22:03:49Z

@daxian-dbw Can you update your review?

src/System.Management.Automation/engine/NativeCommandParameterBinder.cs

src/System.Management.Automation/engine/InitialSessionState.cs

iSazonov · 2021-03-31T18:36:48Z

@mklement0 Do you want to review the PR?

…Binder.cs Co-authored-by: Ilya <darpa@yandex.ru>

TravisEz13 · 2021-03-31T19:10:48Z

@PoshChan Please remind me in 1 hour

PoshChan · 2021-03-31T20:11:06Z

@TravisEz13, this is the reminder you requested 1 hour ago

iSazonov · 2021-04-01T13:33:31Z

Since this PR introduces a breaking change anyway and is being implemented as an experimental feature, I would like to understand why don't we implement mklement0's proposal in full as an experimental feature and close the long history?
See:

Also I asked previously and still wonder why @mklement0 is not in a PowerShell Committee. I believe MSFT team would have saved a lot of time and effort in this case.

TravisEz13 · 2021-04-01T20:30:35Z

@iSazonov can you open a new issue w.r.t. #14692 (comment) and mark it for committee review?

TravisEz13 · 2021-04-01T20:31:20Z

@PoshChan Please remind me in 1 Hour

mklement0 · 2021-04-01T20:36:13Z

The (hopefully) final proposal is now in #14747 (comment)

PoshChan · 2021-04-01T21:31:40Z

@TravisEz13, this is the reminder you requested 1 Hour ago

msftbot · 2021-04-14T23:59:47Z

🎉v7.2.0-preview.5 has been released which incorporates this pull request.🎉

Handy links:

Release Notes

@davidBar-On

[7.2.0-preview.5] - 2021-04-14 * Breaking Changes - Make PowerShell Linux deb and RPM packages universal (#15109) - Enforce AppLocker Deny configuration before Execution Policy Bypass configuration (#15035) - Disallow mixed dash and slash in command line parameter prefix (#15142) (Thanks @davidBar-On!) * Experimental Features - `PSNativeCommandArgumentPassing`: Use `ArgumentList` for native executable invocation (breaking change) (#14692) * Engine Updates and Fixes - Add `IArgumentCompleterFactory` for parameterized `ArgumentCompleters` (#12605) (Thanks @powercode!) * General Cmdlet Updates and Fixes - Fix SSH remoting connection never finishing with misconfigured endpoint (#15175) - Respect `TERM` and `NO_COLOR` environment variables for `$PSStyle` rendering (#14969) - Use `ProgressView.Classic` when Virtual Terminal is not supported (#15048) - Fix `Get-Counter` issue with `-Computer` parameter (#15166) (Thanks @krishnayalavarthi!) - Fix redundant iteration while splitting lines (#14851) (Thanks @hez2010!) - Enhance `Remove-Item -Recurse` to work with OneDrive (#14902) (Thanks @iSazonov!) - Change minimum depth to 0 for `ConvertTo-Json` (#14830) (Thanks @kvprasoon!) - Allow `Set-Clipboard` to accept empty string (#14579) - Turn on and off `DECCKM` to modify keyboard mode for Unix native commands to work correctly (#14943) - Fall back to `CopyAndDelete()` when `MoveTo()` fails due to an `IOException` (#15077) * Code Cleanup <details> <summary> We thank the following contributors! @xtqqczze, @iSazonov, @ZhiZe-ZG </summary> <ul> <li>Update .NET to <code>6.0.0-preview.3</code> (#15221)</li> <li>Add space before comma to hosting test to fix error reported by <code>SA1001</code> (#15224)</li> <li>Add <code>SecureStringHelper.FromPlainTextString</code> helper method for efficient secure string creation (#14124) (Thanks @xtqqczze!)</li> <li>Use static lambda keyword (#15154) (Thanks @iSazonov!)</li> <li>Remove unnecessary <code>Array</code> -> <code>List</code> -> <code>Array</code> conversion in <code>ProcessBaseCommand.AllProcesses</code> (#15052) (Thanks @xtqqczze!)</li> <li>Standardize grammar comments in Parser.cs (#15114) (Thanks @ZhiZe-ZG!)</li> <li>Enable <code>SA1001</code>: Commas should be spaced correctly (#14171) (Thanks @xtqqczze!)</li> <li>Refactor <code>MultipleServiceCommandBase.AllServices</code> (#15053) (Thanks @xtqqczze!)</li> </ul> </details> * Tools - Use Unix line endings for shell scripts (#15180) (Thanks @xtqqczze!) * Tests - Add the missing tag in Host Utilities tests (#14983) - Update `copy-props` version in `package.json` (#15124) * Build and Packaging Improvements <details> <summary> We thank the following contributors! @JustinGrote </summary> <ul> <li>Fix <code>yarn-lock</code> for <code>copy-props</code> (#15225)</li> <li>Make package validation regex accept universal Linux packages (#15226)</li> <li>Bump NJsonSchema from 10.4.0 to 10.4.1 (#15190)</li> <li>Make MSI and EXE signing always copy to fix daily build (#15191)</li> <li>Sign internals of EXE package so that it works correctly when signed (#15132)</li> <li>Bump Microsoft.NET.Test.Sdk from 16.9.1 to 16.9.4 (#15141)</li> <li>Update daily release tag format to work with new Microsoft Update work (#15164)</li> <li>Feature: Add Ubuntu 20.04 Support to install-powershell.sh (#15095) (Thanks @JustinGrote!)</li> <li>Treat rebuild branches like release branches (#15099)</li> <li>Update WiX to 3.11.2 (#15097)</li> <li>Bump NJsonSchema from 10.3.11 to 10.4.0 (#15092)</li> <li>Allow patching of preview releases (#15074)</li> <li>Bump Newtonsoft.Json from 12.0.3 to 13.0.1 (#15084, #15085)</li> <li>Update the <code>minSize</code> build package filter to be explicit (#15055)</li> <li>Bump NJsonSchema from 10.3.10 to 10.3.11 (#14965)</li> </ul> </details> * Documentation and Help Content - Merge `7.2.0-preview.4` changes to master (#15056) - Update `README` and `metadata.json` (#15046) - Fix broken links for `dotnet` CLI (#14937) [7.2.0-preview.5]: v7.2.0-preview.4...v7.2.0-preview.5

msftbot bot assigned TravisEz13 Feb 2, 2021

JamesWTruher added CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log Review - Committee The PR/Issue needs a review from the PowerShell Committee WG-Engine-ParameterBinder labels Feb 2, 2021

mklement0 mentioned this pull request Feb 3, 2021

Arguments for external executables aren't correctly escaped #1995

Closed

JamesWTruher mentioned this pull request Feb 9, 2021

Use ArgumentList when invoking native executables #14747

Open

mklement0 mentioned this pull request Feb 17, 2021

Cmdlet.ThrowTerminatingError() is not affected by -ErrorAction #14819

Open

JamesWTruher force-pushed the NativeInvokeWithArglist01 branch from 7f6581d to dcb79f5 Compare Feb 18, 2021

Steg17 approved these changes Feb 18, 2021

View changes

joeyaiello mentioned this pull request Mar 2, 2021

Create RFC####-Improve-generation-of-argument-string-for-executables PowerShell/PowerShell-RFC#90

Closed

SteveL-MSFT added Committee-Reviewed PS-Committee has reviewed this and made a decision and removed Review - Committee The PR/Issue needs a review from the PowerShell Committee labels Mar 3, 2021

JamesWTruher force-pushed the NativeInvokeWithArglist01 branch from dcb79f5 to 0985d70 Compare Mar 4, 2021

SteveL-MSFT requested changes Mar 4, 2021

View changes

msftbot bot added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Mar 4, 2021

JamesWTruher force-pushed the NativeInvokeWithArglist01 branch from eb105bd to 337877e Compare Mar 9, 2021

msftbot bot removed the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Mar 10, 2021

daxian-dbw reviewed Mar 10, 2021

View changes

src/System.Management.Automation/engine/InitialSessionState.cs Outdated Show resolved Hide resolved

JamesWTruher force-pushed the NativeInvokeWithArglist01 branch from da24160 to e4f7558 Compare Mar 11, 2021

JamesWTruher changed the title ~~WIP: Native invocation using ArgumentList~~ Native invocation using ArgumentList Mar 17, 2021