Testing / CI On Windows With AppVeyor


As the support for Windows begins to mature, I find myself at the juncture where I would like to make sure my package works on Windows and then ensure it continues to work.

In a similar vein to Testing with Travis, it would be good to document a procedure for setting up a CI build for your Package on a Windows machine. AppVeyor is a good option (Travis does not support Windows as a platform yet) and is free for open source repos.

I thought it might be worth working together in this thread to document a process for establishing a Windows build for your package that will run your specs (and allow you to add yet another badge to your README.md ;)).

You can watch along here as I try to figure out how to make this work (help greatly appreciated!): https://ci.appveyor.com/project/joefitzgerald/go-plus

Testing / CI For Packages On Linux
Salesforce.com IDE for Atom (Proton IDE)

Yea the recent official support for Windows has brought quite the influx of new users, judging from the activity on this forum alone.

Looking forward to see if you can get this working, and also how good AppVeyor is in general.


So I made some pretty decent progress on this front yesterday:

Local Run:

c:\projects\go-plus>apm test --path C:\ProgramData\chocolatey\lib\Atom.0.113.0\tools\Atom\Atom.exe
[664:0711/074050:ERROR:gpu_info_collector_win.cc(102)] Can't retrieve a valid WinSAT assessment.
[3232:0711/074050:INFO:renderer_main.cc(227)] Renderer process started
warning: removing Breakpad handler out of order

Finished in 5.5 seconds
35 tests, 177 assertions, 0 failures, 0 skipped

Tests passed

CI Run: https://ci.appveyor.com/project/joefitzgerald/go-plus/build/15

apm test --path C:\ProgramData\chocolatey\lib\Atom.0.113.0\tools\Atom\atom.exe
Command exited with code 1

Is there a way to run with additional logging so that I might debug things on the CI slave?


Hm, I have no idea about any of this… but just looking at what’s going on in that build output, Should you maybe test for this atom.bat instead:

Seems to me that chocolatey is hacking this installation quite a bit.

Edit: Just a speculation, but maybe Choco uses the custom atom.bat in order to abstract from specific versions in the path, which was one of your concerns? Might be an advantage actually, if that’s what it’s doing.


Yea, so looking at that output a bit closer, I’m pretty sure C:\Chocolatey\bin is where Chocolatey creates pseudo executables, pointing at C:\Chocolatey\lib\[program name]\tools\[Atom\atom.exe] in this case… certainly not under C:\ProgramData.

Check out those paths in your build log:


So in the past when I have installed Chocolatey, it has installed to C:\Chocolatey; but when I installed it yesterday it ended up putting itself in C:\ProgramData\Chocolatey on Windows Server 2012 R2.

I wouldn’t get too worried about C:\Chocolatey vs C:\ProgramData\Chocolatey - I think it’s more relevant to try the .bat instead of the .exe directly.

Try #1:

c:\projects\go-plus>apm test --path C:\ProgramData\chocolatey\bin\atom.exe
Tests failed

Note: no .bat file (only an .exe):

c:\projects\go-plus>dir c:\ProgramData\chocolatey\bin
 Volume in drive C is Windows 2012 R2
 Volume Serial Number is 1A82-CE57

 Directory of c:\ProgramData\chocolatey\bin

07/10/2014  02:00 PM    <DIR>          .
07/10/2014  02:00 PM    <DIR>          ..
07/10/2014  02:00 PM               162 apm.bat
07/10/2014  02:00 PM            12,800 atom.exe
07/07/2014  08:25 AM            11,264 choco.exe
07/07/2014  08:25 AM            11,264 chocolatey.exe
07/07/2014  08:25 AM            11,264 cinst.exe
07/07/2014  08:25 AM            11,264 clist.exe
07/07/2014  08:25 AM            11,264 cpack.exe
07/07/2014  08:25 AM            11,264 cpush.exe
07/07/2014  08:25 AM            11,264 cuninst.exe
07/07/2014  08:25 AM            11,264 cup.exe
07/07/2014  08:25 AM            11,264 cver.exe
07/10/2014  01:58 PM            13,312 node.exe
07/07/2014  08:25 AM             1,858 RefreshEnv.cmd
07/10/2014  01:57 PM                21 _processed.txt
              14 File(s)        129,529 bytes
               2 Dir(s)  50,008,035,328 bytes free

This does work locally, but not on the CI server:

c:\projects\go-plus>apm test --path C:\ProgramData\chocolatey\lib\Atom.0.113.0\tools\Atom\Atom.exe
[3740:0711/094328:ERROR:gpu_info_collector_win.cc(102)] Can't retrieve a valid WinSAT assessment.
[3504:0711/094329:INFO:renderer_main.cc(227)] Renderer process started
warning: removing Breakpad handler out of order

Finished in 5.813 seconds
35 tests, 177 assertions, 0 failures, 0 skipped


@ProbablyCorey, @kevinsawicki thoughts on this?


You might want to try the approach that Atom’s build follows when launching the package specs. We don’t use AppVeyor for our Windows build yet but we do have CI for package specs that run as part of the main Windows CI build.


Actually, @batjko you are right - on AppVeyor, Chocolatey installs to C:\Chocolatey (C:\Chocolatey\bin\apm.bat) - it’s very strange that it’s different between my VM (built using https://github.com/joefitzgerald/packer-windows) and that.

Think this might fix it.


I’m an Application Support guy… we follow the error messages wherever they may lead :wink:

Good luck with the build! It seems the coffee spec based approach @kevinsawicki linked could use a bit of simplification via AppVeyor if that works well.


Just a heads up, the Chocolatey install path changed in last week’s release


:checkered_flag: A working solution:

I’ve been using the AppVeyor UI to configure the build; now I need to get appveyor.yml working with a matching configuration and then it should become a reusable approach for others.

This solution is still brittle, because it relies on a versioned path to atom.exe. To play around, I created a file similar to apm.bat called atom.bat at %ALLUSERSPROFILE%\Chocolatey\bin\atom.bat and included this content:

@echo off
SET DIR=%~dp0%
cmd /c "%DIR%..\lib\atom.0.114.0\tools\atom\atom.exe %*"
exit /b %ERRORLEVEL%

Surprisingly, this worked, so it could be a simple fix for the product team to ditch the atom.exe shim and use a .bat file instead. Interestingly, this also fixes the issue where you type atom . and Atom opens an atom directory instead of your current directory.


@kevinsawicki / @nathansobo / @ProbablyCorey: Issue opened with resolution steps provided: https://github.com/atom/chocolatey/issues/23


Here’s a working appveyor.yml file: https://github.com/joefitzgerald/go-plus/blob/d996cb577ca3a96ffcfd9eb486b93e745d033a45/appveyor.yml


Here’s an example AppVeyor build log: https://ci.appveyor.com/project/joefitzgerald/go-plus/build/41 using this appveyor.yml file.

And a nice additional build status badge that would be good to see on other packages soon (Travis is on the left, AppVeyor is on the right):


@kevinsawicki for your reference:

It would be good for someone else to test the appveyor.yml file in https://github.com/atom/ci/pull/3 once https://github.com/atom/apm/pull/149 is merged and a new release of Atom with the changes in it has occurred.


Here’s the first (to my knowledge) example of a package with a Windows CI build guaranteeing the functionality of the package for Windows users: https://atom.io/packages/go-plus.

The release of Atom 0.116.0 means that it is now easy to establish your own CI builds on Windows for your packages.

Go and grab https://github.com/atom/ci/blob/master/appveyor.yml and add it to your repo, then fix all the Windows issues you have with your packages (if any)!