[p-dev] Time to think about CI/testing

[Paul Bone]

I'm not an expert on CI/CD.  I'm fairly sure CD is not what we want.  And
I'm not sure of the precise meaning of CI.  So bear with me if I get
something wrong.

Compiler
========

Plasma has a compiler written in Mercury that has no system specific parts
beyond what Mercury has in its own runtime.  it has a tiny bit of foreign
code in C but it's pretty simple.

We should test with:
 + 32 and 64 bit systems,
 + stable and development versions of Mercury.

Note that development versions of Mercury are currently broken on 32 bit
systems.


Runtime
=======

The rumtime is currently fairly system agnostic, at least until C++ was
added.  It uses a mix of C and C++, and should be tested with at least two
compilers.  There is also a build option to enable extra asserts and other
development options.  

We should test with:
 + 32 and 64 bit systems
 + -DPZ_DEV

If possible:
 + le and be if possible
 + More than one OS (Linux and OS X have been tried so far, if it was known
   to work on something else then I'd add this to "should").
 + Systems that assert for unaligned accesses if possible.
 + valgrind / memcheck / ubcheck / etc.

I'm also in the process of adding a "zealous" mode to the GC, which performs
a lot more checking but runs slower.  And this is why I'm interested in CI.
I don't think we want this to run when people type "make test", not by
default.  But it does make sense to run it for each commit / pull request.

So there's a few things that would be useful to begin testing now.  Or at
the very least to think about testing.  So here's where I throw it over to
you (the reader) since you probably know more about this than me.


New GC mode
===========

So the first question, is for the new GC mode should this be enabled as a
compiler option or an environment variable?  I'm imagining compiling it when
PZ_DEV is defined and enable it when an environment variable is set.  I
imagine that typical CI stuff can handle this but please tell me if it
isn't.


Hosted CI?
==========

When I've worked on other projects, things like "Travis" would run when I
opened a pull request.  Has anyone worked with or set this up before?  Can
it support what we need?

Alternatively I've setup Jenkins before, although I never finished the full
configuration, for a non-hosted thing I think Jenkins seems good.  Does
anyone have any suggestions between hosted/non-hosted and which service to
use?


TAP
===

It was suggested (https://github.com/PlasmaLang/plasma/issues/28) that Test
Anything Protocol (TAP) can be used to gather results from a test suite and
integrate with a CI tool.  Should we use this? is it the best option?  What
other work needs to be done to the test suite / Makefile to make them work
with a CI tool?


I've also posted this as a GH issue: https://github.com/PlasmaLang/plasma/issues/95


Thanks.


-- 
Paul Bone
http://paul.bone.id.au