boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
Hi folks,

It looks like the only reason boostorg/random is not a header-only library
is due to random_device (it is the only src/ file:
https://github.com/boostorg/random/tree/develop/src). The older WinCrypt
calls are not available in some Windows platforms going forward, so this is
going to become a problem pretty quickly for the community.

In the boostorg/uuid library there is a seed_rng class which is doing
similar things (however providing more entropy) and the library is
header-only.  In addition there are two filed defects - one in Trac and one
in github issues, about uuid being incompatible with UWP applications
(that's Windows universal apps).  In uuid, I started adding support for
BCrypt which is a Windows API supported on later versions of Windows, and
once I changed my build flags to target Windows 10 and WINAPI_FAMILY_APP, I
started getting build failures in random_device.

On further inspection it looks like the random library doesn't have support
for BCrypt yet, and even worse as a source library one would need to build
two variants - one for WinCrypt and one for BCrypt to support older and
newer windows platforms.  There are also some implications on MinGW, as the
older original MinGW has support for WinCrypt and MinGW-64 supports both.

I'm going to propose that the implementation from
boost::uuids::detail::seed_rng be moved into boost::random as the new
random_device implementation.  At the same time, boost::random will become
header-only, and thus support setting _WIN32_WINNT at build time of
consuming applications properly, and therefore support new Windows
platforms properly.

At the same time I would like to change seed_rng such that it no longer
silently ignores some return values, and does not disable any compiler
warnings (deprecations, currently).

Thoughts?

Thanks,

- Jim

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
On 12 September 2017 at 15:40, James E. King, III via Boost <
[hidden email]> wrote:

> The older WinCrypt calls are not available in some Windows platforms going
> forward, so this is
> going to become a problem pretty quickly for the community.
>

You worry too much, just call rand_s
<https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx> (the same thing
under the hood, but taking care of any WinCrypt BCrypt, whatever issues).

In the boostorg/uuid library there is a seed_rng class which is doing
> similar things (however providing more entropy) and the library is
> header-only.


The rand_s function uses the operating system to generate cryptographically
secure random numbers. It doesn't get better than that, maybe a call to
RdRand <https://en.wikipedia.org/wiki/RdRand#RDSEED> on a current Intel
CPU, which would use thermal noise (but you'll have to take Intel's word
for it and just cross your fingers they didn't sell you out to a 3-letter
agency).

degski
--
"*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend,
Schmerzen aus Schwäche stillend.*" - Novalis 1798

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
On Tue, Sep 12, 2017 at 4:27 PM, degski via Boost <[hidden email]> wrote:

> On 12 September 2017 at 15:40, James E. King, III via Boost <
> [hidden email]> wrote:
>
>> The older WinCrypt calls are not available in some Windows platforms going
>> forward, so this is
>> going to become a problem pretty quickly for the community.
>>
>
> You worry too much, just call rand_s
> <https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx> (the same thing
> under the hood, but taking care of any WinCrypt BCrypt, whatever issues).

I assume, this is MSVC-specific, i.e. doesn't work with MinGW?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
Part of any effort here would also include taking the Travis and Appveyor
CI files from boost::uuid and run all the checks in boost::random therefore
any incompatibilities such as the one expressed for MinGW would be
identified.  In fact, it would make sense to add some additional unit test
files and/or builds to Appveyor to check different _WIN32_WINNT
compatibility levels to ensure all the build-time code branches are covered
(as well as WINAPI_FAMILY differences).  The documentation on rand_s
doesn't lead me to believe it is compatible with all platforms, as it
expressly states it depends on an API that only works on Windows Desktop
applications, however a well formed CI environment would prove it out if
rand_s was to be used.

- Jim

On Tue, Sep 12, 2017 at 10:13 AM, Andrey Semashev via Boost <
[hidden email]> wrote:

> On Tue, Sep 12, 2017 at 4:27 PM, degski via Boost <[hidden email]>
> wrote:
> > On 12 September 2017 at 15:40, James E. King, III via Boost <
> > [hidden email]> wrote:
> >
> >> The older WinCrypt calls are not available in some Windows platforms
> going
> >> forward, so this is
> >> going to become a problem pretty quickly for the community.
> >>
> >
> > You worry too much, just call rand_s
> > <https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx> (the same thing
> > under the hood, but taking care of any WinCrypt BCrypt, whatever issues).
>
> I assume, this is MSVC-specific, i.e. doesn't work with MinGW?
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/
> mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Andrey Semashev wrote:

> On Tue, Sep 12, 2017 at 4:27 PM, degski via Boost <[hidden email]>
> wrote:
> > On 12 September 2017 at 15:40, James E. King, III via Boost <
> > [hidden email]> wrote:
> >
> >> The older WinCrypt calls are not available in some Windows platforms
> >> going forward, so this is going to become a problem pretty quickly for
> >> the community.
> >>
> >
> > You worry too much, just call rand_s
> > <https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx> (the same thing
> > under the hood, but taking care of any WinCrypt BCrypt, whatever
> > issues).
>
> I assume, this is MSVC-specific, i.e. doesn't work with MinGW?

In addition,

"The rand_s function requires that constant _CRT_RAND_S be defined prior to
the inclusion statement for the function to be declared, as in the following
example:

#define _CRT_RAND_S
#include <stdlib.h>
"

which probably means that if <stdlib.h> is included prior to the header that
wants to use rand_s, it will be unavailable.

We could use RtlGenRandom instead, which is what rand_s calls. Not sure if
that works under UWP.

To answer the question in the original post, yes, making random_device
header-only would indeed be useful, as long as we don't include <windows.h>.

There's no need to include the implementation from seed_rng in its entirety
though, it's more complicated than need be because it tries to support the
case in which a device isn't available. Since the construction of
random_device can fail, it can be a lightweight wrapper over /dev/urandom /
CryptGenRandom, without all the SHA business.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 09/12/17 15:40, James E. King, III via Boost wrote:

> Hi folks,
>
> It looks like the only reason boostorg/random is not a header-only library
> is due to random_device (it is the only src/ file:
> https://github.com/boostorg/random/tree/develop/src). The older WinCrypt
> calls are not available in some Windows platforms going forward, so this is
> going to become a problem pretty quickly for the community.
>
> In the boostorg/uuid library there is a seed_rng class which is doing
> similar things (however providing more entropy) and the library is
> header-only.  In addition there are two filed defects - one in Trac and one
> in github issues, about uuid being incompatible with UWP applications
> (that's Windows universal apps).  In uuid, I started adding support for
> BCrypt which is a Windows API supported on later versions of Windows, and
> once I changed my build flags to target Windows 10 and WINAPI_FAMILY_APP, I
> started getting build failures in random_device.
>
> On further inspection it looks like the random library doesn't have support
> for BCrypt yet, and even worse as a source library one would need to build
> two variants - one for WinCrypt and one for BCrypt to support older and
> newer windows platforms.  There are also some implications on MinGW, as the
> older original MinGW has support for WinCrypt and MinGW-64 supports both.
>
> I'm going to propose that the implementation from
> boost::uuids::detail::seed_rng be moved into boost::random as the new
> random_device implementation.  At the same time, boost::random will become
> header-only, and thus support setting _WIN32_WINNT at build time of
> consuming applications properly, and therefore support new Windows
> platforms properly.
>
> At the same time I would like to change seed_rng such that it no longer
> silently ignores some return values, and does not disable any compiler
> warnings (deprecations, currently).
>
> Thoughts?

I think that making Boost.Random header-only would be good.

In addition I would suggest adding support for getrandom() on Linux and
getentropy() on OpenBSD, as they are protected from file descriptor
exhaustion. This can be done in a separate PR though.

You will likely need to add BCrypt API to Boost.WinAPI - please, make a
PR to it as well.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 12 September 2017 at 17:35, Peter Dimov via Boost <[hidden email]>
wrote:

> ... making random_device header-only would indeed be useful, as long as we
> don't include <windows.h>.
>

 The below works just fine for me, no header:

#include <iostream>

extern "C" void rand_s ( unsigned int* );

int main ( ) {

    unsigned int x;

    rand_s ( &x );

    std::cout << x << '\n';

    return 0;
}

Isn't "C" great (sometimes).

degski
--
"*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend,
Schmerzen aus Schwäche stillend.*" - Novalis 1798

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
degski wrote:

> The below works just fine for me, no header:
>
> #include <iostream>
>
> extern "C" void rand_s ( unsigned int* );

Something like that could probably work, but this is not the correct
prototype. rand_s returns errno_t. If you try

#define _CRT_RAND_S
#include <stdlib.h>

extern "C" void rand_s( unsigned int* );

you get an error.

errno_t seems to be just a typedef for int though.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
On 12 September 2017 at 18:14, Peter Dimov via Boost <[hidden email]>
wrote:

> Something like that could probably work, but this is not the correct
> prototype. rand_s returns errno_t. If you try
>

The code I showed compiles and works, it won't allow you to get the
error-code, though...

So it should (because you are right) be (if one is interested in an error
code):

extern "C" int rand_s ( unsigned int* );


degski
--
"*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend,
Schmerzen aus Schwäche stillend.*" - Novalis 1798

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 9/12/2017 9:27 AM, degski via Boost wrote:

> On 12 September 2017 at 15:40, James E. King, III via Boost <
> [hidden email]> wrote:
>
>> The older WinCrypt calls are not available in some Windows platforms going
>> forward, so this is
>> going to become a problem pretty quickly for the community.
>>
>
> You worry too much, just call rand_s
> <https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx> (the same thing
> under the hood, but taking care of any WinCrypt BCrypt, whatever issues).

Is not rand_s VC++ specific ? There really are other compilers on
Windows other than VC++ ( I can test clang, gcc, and Intel for Windows).

>
> In the boostorg/uuid library there is a seed_rng class which is doing
>> similar things (however providing more entropy) and the library is
>> header-only.
>
>
> The rand_s function uses the operating system to generate cryptographically
> secure random numbers. It doesn't get better than that, maybe a call to
> RdRand <https://en.wikipedia.org/wiki/RdRand#RDSEED> on a current Intel
> CPU, which would use thermal noise (but you'll have to take Intel's word
> for it and just cross your fingers they didn't sell you out to a 3-letter
> agency).
>
> degski
>



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
On 14/09/2017 10:21, Edward Diener wrote:
> On 9/12/2017 9:27 AM, degski wrote:
>
> Is not rand_s VC++ specific ? There really are other compilers on
> Windows other than VC++ ( I can test clang, gcc, and Intel for Windows).

rand_s is MSVCRT specific.  But AFAIK gcc and clang both use MSVCRT on
Windows anyway, so it should work (though might depend on the age of the
CRT used).

If in doubt, you could use RtlGenRandom instead, as Peter Dimov
suggested, since this is the OS-level API.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
RtlGenRandom is no supported on non-desktop UWP, however I'd like to thank
everyone for their responses on this so far.  I'm planning on submitting
some changes to winapi and random, and will look at the existing
random_device as a baseline given feedback on seed_rng, and work towards
making the random library header-only.  I'll be adding some comprehensive
CI integration for travis and appveyor to make sure all the compile-time
branches are being exercised.

On Wed, Sep 13, 2017 at 7:28 PM, Gavin Lambert via Boost <
[hidden email]> wrote:

> On 14/09/2017 10:21, Edward Diener wrote:
>
>> On 9/12/2017 9:27 AM, degski wrote:
>>
>> Is not rand_s VC++ specific ? There really are other compilers on Windows
>> other than VC++ ( I can test clang, gcc, and Intel for Windows).
>>
>
> rand_s is MSVCRT specific.  But AFAIK gcc and clang both use MSVCRT on
> Windows anyway, so it should work (though might depend on the age of the
> CRT used).
>
> If in doubt, you could use RtlGenRandom instead, as Peter Dimov suggested,
> since this is the OS-level API.
>
>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman
> /listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 14 September 2017 at 01:21, Edward Diener via Boost <
[hidden email]> wrote:

> Is not rand_s VC++ specific ? There really are other compilers on Windows
> other than VC++ ( I can test clang, gcc, and Intel for Windows).
>

It's Windows-specific, and has been part of the CRT since Windows XP.

degski
--
"*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend,
Schmerzen aus Schwäche stillend.*" - Novalis 1798

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 14 September 2017 at 02:28, Gavin Lambert via Boost <
[hidden email]> wrote:

> If in doubt, you could use RtlGenRandom instead, as Peter Dimov suggested,
> since this is the OS-level API.
>

rand_s calls RtlGenRandom (or any future API in case that might disappear.

degski
--
"*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend,
Schmerzen aus Schwäche stillend.*" - Novalis 1798

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 14 September 2017 at 03:32, James E. King, III via Boost <
[hidden email]> wrote:

> RtlGenRandom is no supported on non-desktop UWP ...
>

 This page
<https://docs.microsoft.com/en-us/windows/uwp/get-started/whats-a-uwp>
states that: "UWP apps use the Windows Runtime, a native API built into the
operating system." ... and ... Some options for writing apps in UWP
include: ... C++ backend ... So it should be possible to write:

extern "C" int rand_s ( unsigned int* );

which tells the compiler not to worry about the existence of this function,
"we'll give it to you at link-time". This works (and only works AFAIK),
because it's a "C" function, a function without C++ name mangling.

degski
--
"*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend,
Schmerzen aus Schwäche stillend.*" - Novalis 1798

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 09/14/17 03:32, James E. King, III via Boost wrote:
> RtlGenRandom is no supported on non-desktop UWP, however I'd like to thank
> everyone for their responses on this so far.  I'm planning on submitting
> some changes to winapi and random, and will look at the existing
> random_device as a baseline given feedback on seed_rng, and work towards
> making the random library header-only.  I'll be adding some comprehensive
> CI integration for travis and appveyor to make sure all the compile-time
> branches are being exercised.

James, please, don't top post.

http://www.boost.org/community/policy.html#quoting

> On Wed, Sep 13, 2017 at 7:28 PM, Gavin Lambert via Boost <
> [hidden email]> wrote:
>
>> On 14/09/2017 10:21, Edward Diener wrote:
>>
>>> On 9/12/2017 9:27 AM, degski wrote:
>>>
>>> Is not rand_s VC++ specific ? There really are other compilers on Windows
>>> other than VC++ ( I can test clang, gcc, and Intel for Windows).
>>>
>>
>> rand_s is MSVCRT specific.  But AFAIK gcc and clang both use MSVCRT on
>> Windows anyway, so it should work (though might depend on the age of the
>> CRT used).
>>
>> If in doubt, you could use RtlGenRandom instead, as Peter Dimov suggested,
>> since this is the OS-level API.
>>
>>
>>
>> _______________________________________________
>> Unsubscribe & other changes: http://lists.boost.org/mailman
>> /listinfo.cgi/boost
>>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost