Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-23451: Fix socket deprecation warnings in socketmodule.c #2318

Merged
merged 10 commits into from
Jun 28, 2017

Conversation

segevfiner
Copy link
Contributor

@segevfiner segevfiner commented Jun 21, 2017

We are getting this warnings:

Warning	C4996	'inet_addr': Use inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	1067	
Warning	C4996	'WSADuplicateSocketA': Use WSADuplicateSocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	4408	
Warning	C4996	'WSASocketA': Use WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	4648	
Warning	C4996	'WSASocketA': Use WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	4681	
Warning	C4996	'gethostbyname': Use getaddrinfo() or GetAddrInfoW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5119	
Warning	C4996	'gethostbyaddr': Use getnameinfo() or GetNameInfoW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5217	
Warning	C4996	'WSADuplicateSocketA': Use WSADuplicateSocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5347	
Warning	C4996	'WSASocketA': Use WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5350	
Warning	C4996	'inet_addr': Use inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5669	
Warning	C4996	'inet_ntoa': Use inet_ntop() or InetNtop() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5712	
Warning	C4996	'WSAStringToAddressA': Use WSAStringToAddressW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5786	
Warning	C4996	'WSAAddressToStringA': Use WSAAddressToStringW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5925	

I fixed the WSASocket, WSADuplicateSocket warnings by switching to the wide versions. And I added a backwards compatibility note about it. I also fixed WSAAddressToStringA/WSAStringToAddressA by using wchar_t/Py_UNICODE.

inet_ntoa/inet_addr is probably okay in socket_inet_ntoa/socket_inet_aton, but I'm not sure about this one: Modules/socketmodule.c:1067, since it does seem like Windows has inet_pton since Windows Vista: InetPton function (Windows). But it wasn't used for implementing socket_inet_ntop/socket_inet_pton... probably to support older Windows versions. You can also see a Windows specific partial inet_ntop/inet_pton implementation in there, which I don't think is used anymore.

We probably don't want to fix gethostbyname/gethostbyaddr and just ignore it by defining _WINSOCK_DEPRECATED_NO_WARNINGS. I haven't defined it yet so that you can see any remaining warnings. But it should probably be added before this is merged.

The remaining warnings are:

Warning	C4996	'inet_addr': Use inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	1067	
Warning	C4996	'gethostbyname': Use getaddrinfo() or GetAddrInfoW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5119	
Warning	C4996	'gethostbyaddr': Use getnameinfo() or GetNameInfoW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5217	
Warning	C4996	'inet_addr': Use inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5669	
Warning	C4996	'inet_ntoa': Use inet_ntop() or InetNtop() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5712	

EDIT: How should the inet_pton/inet_ntop warnings be handled? Should we go with killing the legacy stuff and the WSAAddressToString/WSAStringToAddress based socket_inet_ntop/socket_inet_pton, and defining HAVE_INET_PTON? or something else? I haven't tested yet whether that actually works... and I don't know what differences in behavior this might cause if any.

cc @zware @zooba

@segevfiner segevfiner changed the title bpo-23451: Fix socket deprecation warnings bpo-23451: Fix socket deprecation warnings in _socketmodule.c Jun 21, 2017
@segevfiner segevfiner changed the title bpo-23451: Fix socket deprecation warnings in _socketmodule.c bpo-23451: Fix socket deprecation warnings in socketmodule.c Jun 21, 2017
@segevfiner
Copy link
Contributor Author

Well I went ahead and used inet_pton/inet_ntop which are available since Windows Vista instead of what was there before, removing the unused code along the way. This also enables the optimization of not calling getaddrinfo in setipaddr for IPv6 on Windows. Which was also where the warning was from.

Were now only left with:

Warning	C4996	'gethostbyname': Use getaddrinfo() or GetAddrInfoW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5112	
Warning	C4996	'gethostbyaddr': Use getnameinfo() or GetNameInfoW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5210	
Warning	C4996	'inet_addr': Use inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5662	
Warning	C4996	'inet_ntoa': Use inet_ntop() or InetNtop() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings	_socket	C:\Users\Segev\prj\python\cpython\Modules\socketmodule.c	5705	

And those are all from wrapper functions of the "deprecated" functions. So the only thing left to do is to define _WINSOCK_DEPRECATED_NO_WARNINGS. Which I will do once requested by whoever reviews this. 😃

Copy link
Member

@zooba zooba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, two concerns:

  • can you try some of the networking libraries test suites (e.g. Twisted, requests, etc.)? I would hope that the stdlib socket tests are sufficient, but I'm not confident
  • rather than adding the preprocessor definition to suppress warnings, can you try #pragma warning first? That way we won't (shouldn't) accidentally introduce new uses of these functions elsewhere.

@@ -60,6 +60,9 @@
<_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
</PropertyGroup>
<ItemDefinitionGroup>
<ClCompile>
<PreprocessorDefinitions>HAVE_INET_PTON;%(PreprocessorDefinitions)</PreprocessorDefinitions>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be defined in PC/pyconfig.h rather than in the project file.

@segevfiner
Copy link
Contributor Author

segevfiner commented Jun 23, 2017

@zooba Doing stuff like:

#ifdef _MSC_VER
#pragma warning(push)
#pragma warning(disable: 4996)
#endif
    h = gethostbyname(name);
#ifdef _MSC_VER
#pragma warning(pop)
#endif

Is a bit ugly... Especially in functions already ridden with preprocessor stuff.

VS2015 does support #pragma warning(suppress) for normal warnings (Used to only be supported for static analysis warnings) But I can't use it with #ifdef surrounding it since it will apply to the #endif line...

There is __pragma (https://msdn.microsoft.com/en-us/library/d9x1s805.aspx) which can be used to define stuff like:

#define MSVC_WARNING_PUSH __pragma(warning(push))
#define MSVC_WARNING_DISABLE(warns) __pragma(warning(disable: warns))
#define MSVC_WARNING_POP __pragma(warning(pop))
#define MSVC_WARNING_SUPPRESS(warns) __pragma(warning(suppress: warns))

Which can make this prettier, but I'm not sure if it's right to add something like this or where to add it to CPython's sources. (pyport.h?)

@segevfiner
Copy link
Contributor Author

The requests test suite passed.

I failed to run Twisted's because it uses tox/virtualenv which explodes when invoked against a built Python source tree (Can't find _socket, likely it doesn't know how to set the path correctly). I'm not sure how to get it to work, maybe there is some script available that can produce a portable Python installation folder so that it has the standard file layout that virtualenv expects?

Do note that I don't have an IPv6 network (IPv6 DNS does work here though). So hopefully they are not skipping those tests 😛

@zooba
Copy link
Member

zooba commented Jun 23, 2017

You should be able to run tools/msi/build.bat to get an installer (will end up in PCBuild/<arch>/en-us/), and this can be installed alongside your other installs without conflicting (it shouldn't include the launcher, which would conflict, but you can deselect that).

As for the warnings, you can probably do something like this at the top of this source file:

#if MS_WINDOWS
#define BEGIN_ALLOW_DEPRECATED_CALL __pragma(warning(push))\
    __pragma(warning(disable: 4996))
#define END_ALLOW_DEPRECATED_CALL __pragma(warning(pop))
#else
#define BEGIN_ALLOW_DEPRECATED_CALL
#define END_ALLOW_DEPRECATED_CALL
#end

There might even be something useful for the #else case, but I don't know what that is. Until we need this elsewhere in the code, there's no harm in only having it in this source file.

@segevfiner
Copy link
Contributor Author

@zooba Oh my... I got hit by unrelated issues trying to do this... 😣

  1. Fix exception in Python 3.7 due to bad re.sub replacement escapes pytest-dev/py#125
  2. setup.py fails under some conditions due to encoding issues kiorky/SOAPpy#26
  3. It seems to have an undeclared dependency on PyWin32. Yay for no PyWin32 wheels (It has a post install script which I think is the major reason it doesn't have any). Luckily they do have builds for Python 3.7 that I could install manually using easy_install.

Had to manually modify the virtualenv created by Tox to workaround these.

BTW, I also found Scripts/msi/make_zip.py which also worked if I modified it to also include .py source files.

I got the following (minus SKIPPED tests):

[FAIL]
Traceback (most recent call last):
  File "twisted\build\py-tests\lib\site-packages\twisted\web\test\test_http.py", line 1832, in test_multipartFormData
    b"HTTP/1.0 200 OK\r\n\r\ndone")
  File "twisted\build\py-tests\lib\site-packages\twisted\trial\_synctest.py", line 432, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "twisted\py37\Lib\unittest\case.py", line 829, in assertEqual
    assertion_func(first, second, msg=msg)
  File "twisted\py37\Lib\unittest\case.py", line 822, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: b'HTTP/1.1 400 Bad Request\r\n\r\n' != b'HTTP/1.0 200 OK\r\n\r\ndone'

twisted.web.test.test_http.ParsingTests.test_multipartFormData
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\test\test_file.py", line 125, in test_timeFormat
    observer(dict(log_format=u"XYZZY", log_time=1.23456))
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\_file.py", line 50, in __call__
    text = self.formatEvent(event)
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\_file.py", line 83, in formatEvent
    event, formatTime=lambda e: formatTime(e, timeFormat)
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\_format.py", line 203, in formatEventAsClassicLogText
    timeStamp = formatTime(event.get("log_time", None))
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\_file.py", line 83, in <lambda>
    event, formatTime=lambda e: formatTime(e, timeFormat)
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\_format.py", line 133, in formatTime
    tz = FixedOffsetTimeZone.fromLocalTimeStamp(when)
  File "twisted\build\py-tests\lib\site-packages\twisted\python\_tzhelper.py", line 86, in fromLocalTimeStamp
    datetime.fromtimestamp(timeStamp) -
builtins.OSError: [Errno 22] Invalid argument

twisted.logger.test.test_file.TextFileLogObserverTests.test_timeFormat
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\test\test_format.py", line 247, in test_formatTimePercentF
    self.assertEqual(formatTime(1.23456, timeFormat="%f"), u"234560")
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\_format.py", line 133, in formatTime
    tz = FixedOffsetTimeZone.fromLocalTimeStamp(when)
  File "twisted\build\py-tests\lib\site-packages\twisted\python\_tzhelper.py", line 86, in fromLocalTimeStamp
    datetime.fromtimestamp(timeStamp) -
builtins.OSError: [Errno 22] Invalid argument

twisted.logger.test.test_format.TimeFormattingTests.test_formatTimePercentF
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "twisted\build\py-tests\lib\site-packages\twisted\names\test\test_server.py", line 1121, in test_sendReplyLoggingNoAnswers
    address=None)
  File "twisted\build\py-tests\lib\site-packages\twisted\names\test\test_server.py", line 224, in assertLogMessage
    [m['message'][0] for m in loggedMessages],
  File "twisted\build\py-tests\lib\site-packages\twisted\names\test\test_server.py", line 224, in <listcomp>
    [m['message'][0] for m in loggedMessages],
builtins.IndexError: tuple index out of range

twisted.names.test.test_server.DNSServerFactoryTests.test_sendReplyLoggingNoAnswers
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\_observer.py", line 131, in __call__
    observer(event)
  File "twisted\build\py-tests\lib\site-packages\twisted\logger\_legacy.py", line 93, in __call__
    self.legacyObserver(event)
  File "twisted\build\py-tests\lib\site-packages\twisted\python\log.py", line 555, in emit
    timeStr = self.formatTime(eventDict["time"])
  File "twisted\build\py-tests\lib\site-packages\twisted\python\log.py", line 530, in formatTime
    tzOffset = -self.getTimezoneOffset(when)
  File "twisted\build\py-tests\lib\site-packages\twisted\python\log.py", line 508, in getTimezoneOffset
    offset = datetime.utcfromtimestamp(when) - datetime.fromtimestamp(when)
builtins.OSError: [Errno 22] Invalid argument

twisted.names.test.test_server.DNSServerFactoryTests.test_sendReplyLoggingNoAnswers
twisted.names.test.test_server.DNSServerFactoryTests.test_sendReplyLoggingNoAnswers
twisted.names.test.test_server.DNSServerFactoryTests.test_sendReplyLoggingWithAnswers
twisted.names.test.test_server.DNSServerFactoryTests.test_sendReplyLoggingWithAnswers
twisted.names.test.test_server.DNSServerFactoryTests.test_sendReplyLoggingWithAnswers
twisted.names.test.test_server.DNSServerFactoryTests.test_sendReplyLoggingWithAnswers
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "twisted\build\py-tests\lib\site-packages\twisted\names\test\test_server.py", line 1146, in test_sendReplyLoggingWithAnswers
    address=None)
  File "twisted\build\py-tests\lib\site-packages\twisted\names\test\test_server.py", line 224, in assertLogMessage
    [m['message'][0] for m in loggedMessages],
  File "twisted\build\py-tests\lib\site-packages\twisted\names\test\test_server.py", line 224, in <listcomp>
    [m['message'][0] for m in loggedMessages],
builtins.IndexError: tuple index out of range

twisted.names.test.test_server.DNSServerFactoryTests.test_sendReplyLoggingWithAnswers
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "twisted\build\py-tests\lib\site-packages\twisted\python\test\test_zipstream.py", line 253, in test_extraData
    with zipstream.ChunkingZipFile(fn) as czf, czf.readfile("0") as zfe:
  File "twisted\py37\Lib\zipfile.py", line 1107, in __init__
    self._RealGetContents()
  File "twisted\py37\Lib\zipfile.py", line 1230, in _RealGetContents
    x._decodeExtra()
  File "twisted\py37\Lib\zipfile.py", line 442, in _decodeExtra
    raise BadZipFile("Corrupt extra field %04x (size=%d)" % (tp, ln))
zipfile.BadZipFile: Corrupt extra field 6568 (size=27756)

twisted.python.test.test_zipstream.ZipstreamTests.test_extraData
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "twisted\build\py-tests\lib\site-packages\twisted\test\test_log.py", line 564, in test_microsecondTimestampFormatting
    self.assertEqual("600000", self.flo.formatTime(12345.6))
  File "twisted\build\py-tests\lib\site-packages\twisted\python\log.py", line 528, in formatTime
    return datetime.fromtimestamp(when).strftime(self.timeFormat)
builtins.OSError: [Errno 22] Invalid argument

twisted.test.test_log.FileObserverTests.test_microsecondTimestampFormatting

None of them seem related to this change.

@zooba
Copy link
Member

zooba commented Jun 26, 2017

@segevfiner Yeouch, those aren't good.

The random corruption does concern me a little bit - I don't want to presume what's related to the change here. If they occur without the fix, then let me know and I'd say we're done.

Thanks for working through these! It's much appreciated :)

@segevfiner
Copy link
Contributor Author

@zooba I get those same failures with a build from master.

@zooba
Copy link
Member

zooba commented Jun 26, 2017

Great. The last thing needed is the NEWS entry, and it looks like you'll get to be one of the first to use the new system :)

See https://docs.python.org/devguide/committing.html#what-s-new-and-news-entries for info - it's basically just creating a single file in the Windows directory with the title of this bug (and add "Patch by " to ensure you get proper credit).

@zooba
Copy link
Member

zooba commented Jun 28, 2017

I fixed the conflict, and if the checks pass then nothing left to do and I'll merge. Thanks!

@zooba zooba merged commit 9f3bdcb into python:master Jun 28, 2017
@zooba
Copy link
Member

zooba commented Jun 28, 2017

I'll try, but anyone can feel free to keep an eye on http://buildbot.python.org/all/waterfall?category=3.x.stable to see if any builds have issues as a result of this (or if the warnings still appear - we should have significantly less thanks to this patch).

@zooba
Copy link
Member

zooba commented Jun 28, 2017

@segevfiner
Copy link
Contributor Author

@zooba BTW #2375 should fix the last warnings on 32-bit Windows. 😉

@segevfiner segevfiner deleted the bpo-23451-socket-deprecation-warnings branch June 28, 2017 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants