As @Jim Foye pointed out, the best way to go is to reference the package with #r "nuget:".
As for why your original code throws an exception: the assemblies in the ref folder of a package, when there is one, are so-called "reference assemblies". They contain all the same classes and methods as the corresponding runtime assemblies except with invalid implementations, and they are only meant to be used during compilation, not at runtime. The assemblies that are actually used at runtime are in the lib folder.
