Hello @metaskills,
did your tests against Azure pass lately?
I tried 1.0.5 and AR 4.2 and still get the same problem. First connect fails, seconds works then. Also to mention that the 1.0.2g open-ssl still has this CPU bug that loading rails console takes 5 times compared to the fixed libeay32.dll.

bye Yves

@metaskills
I am currently experiencing the TinyTDS Not valid win32 Application error when executing a chef run. In some contexts, I have been able to resolve the problem by requiring TinyTDS in a particular order with the other requirements that use the conflicting components, however I have encountered a situation where this is not sufficient, I still get the error. I attempted to use chcp to resolve the issue as you suggested, however I am unsure what is meant by setting it up "for" a file. Is that different than running it in the same powershell window where tinyTDS will be used? In any case, chcp 866 was used in said powershell window, chef was executed, and the error still occurred. Any information you can provide would be appreciated.

So I've been poking at this on and off for around a day now, and I can't seem to get around the version constraints of the DevKit environment which prevent OpenSSL 1.1.x from building. My next step would be to move the responsibility of building OpenSSL outside of devkit (so not a part of the rake script), using the build environments available on appveyor, and finally consuming the built library within the tiny_tds build.

What do you guys think, is there any reason for me not to pursue this option? I'm pretty new to these concerns in windows and DevKit so I welcome any suggestions.

As an aside are there any reasons not to use one of the pre-built openssl libraries for windows mentioned on the openssl wiki? https://wiki.openssl.org/index.php/Binaries

@YvesR Yes, I've run the Azure tests and they are fine. But I am mostly focusing on SQL Server on Linux and hence the latest work in master for it and CircleCI.

@nmscholl That is the symptom and reason for this issue. The fix that I am proposing is to use OpenSSL v1.1.x but that is not so easy and it requires work that I have not prioritized. @YvesR is suggesting to drop in a different DLL. Both require some work that I have not yet had time to prioritize.

@coderjoe Neat idea!!! Can we?

@metaskills honestly, not sure! But I'll keep poking at it. No promises! ;)

@metaskills I have a test env where I do db access directly on start with a initialize script. This fail to start. Starting a console without first db access fails, next and following ones success.

About open SSL 1.1.x: Is it compatible? I read that the openSSL team broke up some compatibilities. So we can just use latest version what @coderjoe suggests?

@YvesR well my suggestions only really make sense if we're going to OpenSSL 1.1.x... if we keep following the LTS branch (1.0.2x) then it doesn't make sense to change the build system because it shouldn't be necessary yet.

The LTS support branch (1.0.2x) is good until December 31st 2019 so there's no immediate need to switch unless it's the only thing that fixes the binary incompatibility problem on windows. It would be much easier if switching to 1.0.2j fixed it, but I can't get the appveyor tests to pass there. (I notice you're having a similar problem @metaskills in your recent PR)

Honestly 1.1.x or 1.0.2x doesn't matter to me, I'm just desperate to solve this invalid windows binary problem since it's a critical blocker in my work.

It would be much easier if switching to 1.0.2j fixed it
I notice you're having a similar problem @metaskills in your recent PR

Yup! See the issue description about C:\Perl\bin and such. I have yet to address this. Is there anything I can do to get help with that?

I meant more in PR #320 , before I started on the OpenSSL 1.1.x path I tried 1.0.2j but ran into the same build problem you are, which seems independent of the Perl version since 1.0.2x doesn't require updated Perl.

As for help, nothing I can think of now, I'm still learning the toolchain and experimenting with permutations so I don't know enough yet to help me ask the right questions. Do you know of any good guides/resources on setting up test VMs to test/verify tiny_tds on windows? Right now I'm working off of a Windows 10 Evaluation VM I set up to help me out.

Well my main concern about openSSL is the bug on windows that it causes 100% CPU usage for no reason. A rails console starts in like 4 seconds on my dev machine. With 1.0.2.x openSSL on windows, which still has the bug, it needs like 15 seconds with big CPU usage.

@YvesR I just spent a good part of a day hacking on this trying to get it to work. Have you verified that OpenSSL 1.1.0 is supported by freetds? From everything I can tell it relies on direct struct access to BIO information in SSL which isn't accessible anymore in OpenSSL 1.1.0. It also relies on defines that no longer exist in 1.1.0.

I'm starting to think OpenSSL 1.1.0 just is not an option until upstream sources start building for it.

@coderjoe I never dig so deep to check about FreeTDS itself. I tried to build tinyTDS on ym dev machine, but failed to modify the build task with latest OpenSSL build. In #250 you can read that we manually replaced the libeay32.dll. I took the DLL provided by @lukajandric and manually replaced it in tinyTDS 0.7.0. That did not work in a 0.9.x release but works again in 1.0.5. So I work with the latest 1.0.5 gem and manually replaced the libeay32.dll there.

@YvesR Can you post a link to that DLL? Would we be able to add that to the repo here and cp it over during build time for Windows till the longer 1.1 plan can be worked on?

Yes, sure, it was posted first time in #250.
But here again: libeay32.zip
Patch Info

FWIW, after reading the OpenSSL patch info, it would seem like they aren't patching OpenSSL because Microsoft released a hotfix. In my opinion, installing the hotfix on affected machines is preferable to maintaining an unsupported/patched version of OpenSSL which could open the project up to other bugs.

@YvesR would this hotfix address your problem?

https://support.microsoft.com/en-us/kb/2719306

@coderjoe sadly no. The bug is not fixed by Microsoft. It still happens on Windows 2012 R2 and Windows 2016! I run 2012 machines, not 2008 anymore. Dev 2016 machines I run have same issue, too.
So in fact the issue is solved by the lib from the link I posted and not fixed without on any current windows machine.

@YvesR that's extremely unfortunate. Have you opened an incident with Microsoft? You might have to.

Looking at the patch it would seem the patch may be dangerous. It seems to suggest that by moving from Heap32First and Heap32Last to HeapWalk that you limit the heaps being walked to only the current process's heap. That seems like a tremendously bad idea since the heapwalk is helping seed the random number generator with entropy...

To quote the second post in the patch info discussion:

But the RAND code in openSSL is using the heap walking to get as many
random allocation details as possible from all processes in the system
to seed its RNG.
So limiting the RAND code to only a single heap from its own process
will effectively make that code useless and severely weaken the security
of all cryptographic keys and nonces produced by openSSL. It is simply
not an option.

http://openssl.6102.n7.nabble.com/Deadlock-in-RAND-poll-s-Heap32First-call-td13043.html

I would caution against using the patched binary. Speeding up startup by a few seconds is not worth compromising OpenSSL's entropy.

@YvesR as an aside, how large is the combined heap on your machines? I haven't been experiencing these slow startups and I'm wondering if it's just because of a smaller system heap. Could it be it is taking so long because there is a lot of heap to walk and that's just how long it takes?

The bug the patch fixes is not a slow startup, it's a deadlock which would imply you are not hitting that bug if your application eventually starts.

@coderjoe My application which has the problem has some config/intializer scripts where I read data from database at startup. That is probably the reason why I run into this bug from start.

I run a dev machine on my MacBook on VMware Fusion which runs Windows 2012 R2 with 4 Cores and 8GB RAM.

• Rails v4.2

• Ruby 2.0.0p598

• tinyTDS 1.0.5

• activerecord-sqlserver-adapter 4.2.15

• Using not fixed DLL => 10,63 seconds to load "rails c". While 80% of the time 100% CPU usage on this core that runs the ruby.exe.

• Using fixed DLL => 4,26 seconds while I never hit 100% CPU usage on that core.

I am not experienced that deep but the CPU load and time speaks for itself that this has to be addressed.

ADD sidenote: This dev machine has fast SSD disks. In a "slow" Azure machine it takes >30 seconds...

@YvesR I agree that your problem should be addressed, but I do not believe the patch you have provided is the best way to go about it.

Frankly, I am not comfortable with the patch and its potential security implications, and if you're patching core Ruby with this DLL to speed up load times I would recommend only doing it in development, not in production.

So I've hacked together some changes using OpenSSL 1.0.2j and freetds 1.00.21 which result in binaries which appear to avoid the invalid win32 executable error as well as the error 998: invalid memory access error.

How would you like to go about reviewing this? Just submit a pull request for review? It has some not-so insignificant changes.

@metaskills what are the chances I could get an ETA review on #322? If my changes aren't acceptable I can work towards an alternative.

I don't want to rush you but the invalid win32 executable is a production blocker at work. :(

Just did that now and also added you as a collaborator. Cheers!

Just released v1.1.0. Please test with that.

@metaskills I doubt you address me with the test, but I did anyway :)

• Local system: Works as usual, no error
• Local system with SSL encrypted SQL server: Works without error
• CPU usage on load: Same as before, but you didn't address this with 1.1.0 I guess
• Azure connect: 1st connect fails, 2nd attempt works

1.1.0 has been fine for me so far, no win32 invalid executable errors

Just noticed this commit in FreeTDS (FreeTDS/freetds@bbe6d1b) which says "Provide compatibility with OpenSSL 1.1". So this seems ripe to try again now to move from OpenSSL v1.0 to v1.1

@metaskills In the build scripts we can just change the versions to compile so it fetches the correct sources or does the tiny built process needs to be modified?

Yup, making a PR with that change will tell us if that works. Simple change in extconsts and a few other places. See #322 for reference.

Oh... we may have to wait till FreeTDS has a release... I'd have to check this distills to see what version that has. Likely might be a day or so since that commit just landed.

Ok, let us wait until that and I will try it then. Hopefully they are ready until weekend. Good time for tests :).

FYI, tried to take a look at this today and I noticed the FreeTDS FTP site was down. I sent an email to their mailing list. Will try again in a few days.

Did you manage to compile? I change extconst.rbthis weekend to open ssl 1.1.0e and freetds to 1.0.27 and modified extconf.rbto

    execute "mkdef-libeay32", "(perl util/mkdef.pl 32 crypto >libeay32.def)"
    execute "mkdef-ssleay32", "(perl util/mkdef.pl 32 ssl >ssleay32.def)"

Also update my docker image to rake compile 0.6.0. For testing I compiled with original config. Works fine. With modified settings openssl compile worked but freetds failed to build.

My Logfile: https://gist.github.com/YvesR/098ad10b6222f83d7c4204ef64bfc24c

The branch/pr #344 is where I am working on that. Comments and help there welcome.

PR #374 got a good build (from what I can tell) of OpenSSL 1.1 for Windows. I just published v1.2.0.beta1 from that branch. Please give it a try.

I pulled #347 and did run rake gem:windows

Compile works so far, that is awesome.
But in the final stage to make the package I get this.

mkdir -p pkg/tiny_tds-1.1.0/ports/archives
rm -f pkg/tiny_tds-1.1.0/ports/archives/freetds-1.00.27.tar.bz2
ln ports/archives/freetds-1.00.27.tar.bz2 pkg/tiny_tds-1.1.0/ports/archives/freetds-1.00.27.tar.bz2
cd pkg/tiny_tds-1.1.0
mv tiny_tds-1.1.0.gem ..
cd -
rake aborted!
Don't know how to build task 'ports/i686-w64-mingw32/bin/libcrypto32-1.1.0e-i686-w64-mingw32.dll' (see --tasks)
/usr/local/rvm/gems/ruby-2.4.0/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.4.0/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => gem => pkg/tiny_tds-1.1.0-x86-mingw32.gem
(See full trace by running task with --trace)
rake aborted!
Command failed with status (1): [docker run -v /Users/rausch/Documents/_Projekte/tiny_tds:/Users/rausch/Documents/_Projekte/tiny_tds -e UID\=1000 -e GID\=1000 -e USER\=rausch -e GROUP\=_staff -e ftp_proxy\= -e http_proxy\= -e https_proxy\= -e RCD_HOST_RUBY_PLATFORM\=x86_64-darwin15 -e RCD_HOST_RUBY_VERSION\=2.3.1 -e RCD_IMAGE\=larskanis/rake-compiler-dock:0.6.0 -w /Users/rausch/Documents/_Projekte/tiny_tds --rm -i -t larskanis/rake-compiler-dock:0.6.0 runas sigfw bash -c \ \ \ \ bundle\ \&\&\ rake\ cross\ native\ gem\ RUBY_CC_VERSION\=2.4.0:2.3.0:2.2.2:2.1.6:2.0.0\ CFLAGS\=\"-Wall\"'
']
/Users/rausch/Documents/_Projekte/tiny_tds/Rakefile:91:in `block in <top (required)>'
Tasks: TOP => gem:windows
(See full trace by running task with --trace)

On some point the copy of the compiled stuff do not copy the necessary files.

Can you check the v1.2.0.beta1 gem published to rubygems? Does it exhibit the same behavior?

Do you have the link? when I gem install tiny_tds.1.2.0.beta1.gem I get error that it is not found.

Update My bad, had a certificate problem that needed to be fixed first for rubygems.org on the windows test machine. Got it and will test now.

I would add the version to Bundler. You would need a -v arg if you wanted to target a specific version. I tend to use --pre tho:

$ gem install --pre tiny_tds
Fetching: tiny_tds-1.2.0.beta1.gem (100%)
Building native extensions.  This could take a while...
Successfully installed tiny_tds-1.2.0.beta1
1 gem installed

But again, I would add the version to my Gemfile and bundle.

Yeah, did a modify on my gemfile and bundle install. Worked fine after fix my cert-problem.

And now the good news first or the better news :)?

The good news: rails console opens without high CPU load anymore and much faster then before

The better news: rails works with Azure SQL now without the previous issues

Open champagne, you earned it! :) I will do more tests now and seek more into it...

Huge thanks to @orgads!!!

I'm glad to see it works :)

Pull Request #347 pulled in OpenSSL v1.1.0 and we got new versions of TinyTDS at v1.2.0 which should solve this issue. Please reply back and let us know.