Publish a .csx script as a binary executable #312

Sharpiro · 2018-06-09T23:08:41Z

This PR allows for publishing a script as an executable.

I found myself every now and then having a good use case to basically take some scripts I've created and just have them outputted to an executable file. I thought this would be kind of cool and looked relevant to the current issue: #227. However this is slightly different in that this isn't currently concerned with outputting scripts to re-usable libraries, and just executes as if dotnet script main.csx were called, except from a .exe file. If this isn't something you guys want in the main code base, I understand. I also did my best to modify as few files as possible and follow any coding styles I noticed while learning the code base.

I created a new "publish" command that takes in a script argument, an optional publish directory, and a debug flag. It uses a lot of code that you guys have already written, but goes a bit further with modifying the generated temp project, adding a Program.cs template, generating an assembly from the scripts, and moving around any referenced .dll files as needed. dotnet publish is then executed from your command runner class and this creates the executable.

Let me know if you have any questions or if there is an obvious better way to do this. Thanks.

Example usage:

dotnet script publish C:\Users\sharpiro\Desktop\temp\tempScript\main.csx -d -o C:\Users\sharpiro\Desktop\temp\publishTemp

dotnet executable dll now functional adding temporary project builder solution adding temp script code working on console app ref restore preparing to add dotnet cli tools now outputting to exe with args and printing errors

removed testing data

filipw · 2018-06-11T07:13:51Z

Thanks a lot, this is a great idea, and this is definitely a feature that we want.
At the high level, this implementation is also along the lines of what we'd want to have (compile, reference the DLL and run precompiled code using <Factory> method).

You can have a look at my PoC for #227 which is really very similar, it's here e677cd6

I think what I'd like to see is if you incorporate the ScriptEmitter from that commit; this would decouple the process of emitting from the publish, which will be helpful in the future.

Can you also add some tests? I'll also add some comments inline - thanks!

filipw · 2018-06-11T07:15:34Z

src/Dotnet.Script.Core/ScriptPublisher.cs

+                    .WithScriptClassName(AssemblyName);
+                var scriptCompilation = compilationContext.Script.GetCompilation()
+                      .WithOptions(scriptOptions)
+                    .WithAssemblyName(AssemblyName);


is the assembly name reset really necessary?

I'll check on this and make sure

filipw · 2018-06-11T07:16:55Z

src/Dotnet.Script.Core/ScriptPublisher.cs

+
+            var projectFile = new ProjectFile(File.ReadAllText(tempProjectPath));
+            // todo: grab version in a better way?
+            projectFile.AddPackageReference(new PackageReference("Microsoft.CodeAnalysis.Scripting", "2.8.2", PackageOrigin.ReferenceDirective));


I think it's ok to hardcode a version. You could technically omit it, with .net core SDK 2.1.300 it would just pull the latest but it might conflict with something. Let's make a global constant for the version

filipw · 2018-06-11T07:18:17Z

src/Dotnet.Script.Core/ScriptPublisher.cs

+
+                foreach (var reference in scriptCompilation.DirectiveReferences)
+                {
+                    var refInfo = new FileInfo(reference.Display);


you may want to check for nuget pragma here, if it's a nuget reference there is nothing to copy (it resolves to a fake DLL) so you can skip it.

k I'll add something for this, I noticed what you're talking about

filipw · 2018-06-11T07:19:00Z

src/Dotnet.Script.Core/ScriptPublisher.cs

+                }
+
+                var assemblyPath = $"{tempProjectDirecory}/{AssemblyName}.dll";
+                var emitResult = scriptCompilation.Emit(assemblyPath);


let's incorporate ScriptEmitter instead e677cd6

filipw · 2018-06-11T07:20:04Z

src/Dotnet.Script.Core/Templates/program.publish.template

+                var factoryMethod = typeof(scriptAssembly).GetMethod("<Factory>");
+                if (factoryMethod == null) throw new Exception("couldn't find factory method to initiate script");
+
+                // todo: not sure what the second parameter is, using 'null' for now


this is OK - the second index in the array is for submission continuations (REPL scenarios)

filipw · 2018-06-11T07:20:25Z

src/Dotnet.Script.Core/Templates/program.publish.template

+{
+    class Program
+    {
+        static void Main(string[] args)


you can change it to async Main and then there is no need to block in the code

filipw · 2018-06-11T07:21:09Z

src/Dotnet.Script.Core/Templates/program.publish.template

+                var invokeResult = invokeTask.Result;
+                if (invokeResult != 0) WritePrettyError($"Error result: '{invokeResult}'");
+            }
+            // todo: not getting the full script stack trace


what do you mean?

so i noticed that when i would throw an error in the script, I wouldn't get any of the script stack trace information to print out. I can see if that was because I wasn't using your emitter with debug information maybe?

for example, if i threw an error on line 25 of the main.csx file, when writing out the error it wouldn't give me any indication that the error actually occurred on 25, or if there was previous stack trace data

this is probably because you emit without symbols - if you compare the emitter code I mentioned before, it emits symbols too and line numbers are available in the symbols only

filipw · 2018-06-11T07:21:55Z

src/Dotnet.Script.Core/Templates/program.publish.template

+                if (invokeResult != 0) WritePrettyError($"Error result: '{invokeResult}'");
+            }
+            // todo: not getting the full script stack trace
+            catch (AggregateException ex)


I think you want something like this

if (e is AggregateException aggregateEx) { e = aggregateEx.Flatten().InnerException; }

wouldn't i need to still loop through the flattened "InnerExceptions" of a flattened aggregate exception typically? I think you're right though in this case there will only ever be 1 inner exception in that aggregate exception.

I think on these code paths there can only ever be one

Sharpiro · 2018-06-12T03:00:26Z

Thanks for the feedback. I'll try to update this code to use your script emitter, make adjustments based on your comments, and add a few tests hopefully tomorrow.

adding scriptEmitter

Sharpiro · 2018-06-12T16:16:01Z

I just committed code based on your feedback.

I found that I did indeed need to modify the assembly name since prior to that it was an unspeakable name and would also be an invalid file name. Even when the assembly name was kept the same, but just written to a valid file name, there would be an error where the code would be looking for that unspeakable assembly name.

I'll go ahead and start working on some tests soon.

seesharper · 2018-06-12T16:21:53Z

src/Dotnet.Script.Core/ScriptEmitResult.cs

+        public MemoryStream PdbStream { get; }
+        public ImmutableArray<Diagnostic> Diagnostics { get; private set; } = ImmutableArray.Create<Diagnostic>();
+        public ImmutableArray<MetadataReference> DirectiveReferences { get; } = ImmutableArray.Create<MetadataReference>();
+        public bool IsErrored => Diagnostics.Any();


Do we need any suppressions here?
https://github.com/filipw/dotnet-script/blob/4b72e5186966a1dbbc1d24c6816d85f5d9632457/src/Dotnet.Script.Core/ScriptCompiler.cs#L55

also, diagnostics would also contains warnings. I think you should just check if there are any diagnostics that are error level

Sharpiro · 2018-06-12T17:15:58Z

So how does error suppression work here? I noticed you're suppressing CS1705 which is an error, not a warning like the others. Even though you are supprsssing this error when evaluating the script diagnostics, won't this error just cause the script or emit to fail later on anyways? I tested a different error and tried to suppress it through your process, but it just causes the emit to fail in the script emitter later on.

Also I don't see any current reason to suppress warnings. From what I can see dotnet script has no functionality or perhaps no desire to log or print warnings to the user. It simply logs the diagnostics it suppressed and then errors and logs any unsuppressed errors if any were encountered.

seesharper · 2018-06-12T20:43:50Z

The need for suppressing CS1705 actually went away with #261. We decided to keep it nevertheless in case we still had assembly loading issues.
We are probably okay not suppressing it 👍

seesharper · 2018-06-12T20:46:43Z

src/Dotnet.Script.Core/ScriptEmitResult.cs

+        public MemoryStream PdbStream { get; }
+        public ImmutableArray<Diagnostic> Diagnostics { get; private set; } = ImmutableArray.Create<Diagnostic>();
+        public ImmutableArray<MetadataReference> DirectiveReferences { get; } = ImmutableArray.Create<MetadataReference>();
+        public bool IsErrored => Diagnostics.Any();


Success might be a better name her since that aligns with EmitResult.Success

seesharper · 2018-06-12T20:48:30Z

src/Dotnet.Script.Core/ScriptPublisher.cs

+                    throw new CompilationErrorException("One or more errors occurred when emitting the assembly", emitResult.Diagnostics);
+                }
+
+                var assemblyPath = $"{tempProjectDirecory}/{AssemblyName}.dll";


Use Path.Combine to ensure correctness across all operating systems 👍

seesharper · 2018-06-12T20:49:54Z

Left a few nit picking comments. Otherwise LGTM. Oh and the branch needs updating 😄

filipw · 2018-06-13T09:57:31Z

I noticed you're suppressing CS1705

this has been historically suppressed in the .NET CLI as well as in scripting, because the runtime should redirect to the correct version.
OmniSharp suppresses those too https://github.com/OmniSharp/omnisharp-roslyn/blob/76810f8655fbd198187b8ef0eae8bd6a3bee3c4c/src/OmniSharp.Roslyn/Utilities/CompilationOptionsHelper.cs#L10-L16

suppressing warnings on failure misc pr review changes

filipw

LGTM

seesharper · 2018-06-13T16:56:49Z

@Sharpiro This all looks good although I would love to see some basic testing before merging 😎

added several simple tests

…net-script into feature/console-app

Sharpiro · 2018-06-13T17:52:11Z

just added a few simple tests and am now using absolute file paths in the same way the default app command is.

Sharpiro · 2018-06-13T18:45:02Z

Looks like the Travis CI build didn't like my tests

The tests work on my machine as well as AppVeyor. Anything I need to know about Travis CI?

seesharper · 2018-06-13T19:51:18Z

@Sharpiro Okay, a couple of things here. When compiling a self contained binary on *nix, they don't get the .exe extension. Also I would like to have a test that actually executes the generated binary 👍

testing that generated exectuables execute correctly

seesharper · 2018-06-13T20:55:42Z

Now we are in business 🎸 Thanks 👍

Sharpiro added 5 commits June 9, 2018 18:29

removing unnecessary changes

ab7b66e

reviewed/refactored to modify less files overall

08f8295

adding debug flag for publish

245876a

starting on output to .exe

3f47c70

dotnet executable dll now functional adding temporary project builder solution adding temp script code working on console app ref restore preparing to add dotnet cli tools now outputting to exe with args and printing errors

publisher now works with script assembly references

1acf71f

removed testing data

filipw reviewed Jun 11, 2018

View reviewed changes

pr feedback

2fed7a4

adding scriptEmitter

seesharper reviewed Jun 12, 2018

View reviewed changes

seesharper approved these changes Jun 12, 2018

View reviewed changes

using path.combine for multi-platform

59434cd

suppressing warnings on failure misc pr review changes

filipw approved these changes Jun 13, 2018

View reviewed changes

Merge branch 'master' into feature/console-app

4c44c2b

Sharpiro added 2 commits June 13, 2018 13:43

always using absolute paths for input script and publish directory

a517c4e

added several simple tests

Merge branch 'feature/console-app' of https://github.com/Sharpiro/dot…

81677f4

…net-script into feature/console-app

trying to make tests cross-platform

e858c15

testing that generated exectuables execute correctly

seesharper changed the title ~~Publish a .csx script as .exe~~ Publish a .csx script as a binary executable Jun 13, 2018

seesharper merged commit 2015425 into dotnet-script:master Jun 13, 2018

Sharpiro deleted the feature/console-app branch June 14, 2018 14:24

Sharpiro mentioned this pull request Jun 16, 2018

Feature/create and execute dll #315

Merged

seesharper mentioned this pull request Sep 2, 2018

Add support for outputting into a DLL #227

Closed

Publish a .csx script as a binary executable #312

Publish a .csx script as a binary executable #312

Uh oh!

Conversation

Sharpiro commented Jun 9, 2018

Uh oh!

filipw commented Jun 11, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sharpiro commented Jun 12, 2018

Uh oh!

Sharpiro commented Jun 12, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sharpiro commented Jun 12, 2018

Uh oh!

seesharper commented Jun 12, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seesharper commented Jun 12, 2018

Uh oh!

filipw commented Jun 13, 2018

Uh oh!

filipw left a comment

Choose a reason for hiding this comment

Uh oh!

seesharper commented Jun 13, 2018

Uh oh!

Sharpiro commented Jun 13, 2018

Uh oh!

Sharpiro commented Jun 13, 2018

Uh oh!

seesharper commented Jun 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seesharper commented Jun 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

seesharper commented Jun 13, 2018 •

edited

Loading

seesharper commented Jun 13, 2018 •

edited

Loading