Windows System

Programming using
Python
Mark Hammond
mhammond@skippinet.com.au

OReilly Open Source Python Conference

August 1999, Monterey, CA

About this Presentation

Most

content taken directly from upcoming book for OReilly

Python Programming on Win32

By Mark Hammond and Andy Robinson
http://www.ora.com/catalog/pythonwin32/

OReilly

Windows System Prog

Slide 2

Who this talk is for?

Existing

Python programmers

Even those without any Windows experience should follow this

without problem.
Although

Existing

there is not enough time to explain the relevant Windows APIs

Windows Programmers

Even without Python experience, you should immediately see the

similarities between your existing language.

OReilly

Windows System Prog

Slide 3

What is Python
Interpreted,

dynamic high-level language

Obviously open source
Hence we are here.
Often

used in a similar problem domain to Perl/Tcl

Proponents consider readability and maintainability a big strongpoint
http://www.python.org

OReilly

Windows System Prog

Slide 4

Python and Windows

Each

Python version comes with an installer package for

Windows.
Standard Python port contains all cross-platform Python
features, but very few Windows specific features.
Python for Windows extensions contains many useful Windows
extensions for Python.
http://www.python.org/windows

OReilly

Windows System Prog

Slide 5

Python Windows Extensions

Includes:

Pythonwin: MFC based GUI environment and IDE/Debugger

win32com: Interfaces Python and COM
win32 Extensions: Interfaces to native Win32 API.
Official

releases can be found at http://www.python.org/windows

Extensions home is at http://starship.python.net/crew/mhammond

OReilly

Windows System Prog

Slide 6

System Level Programming?

For

this talk, we define system level programming as

working with low-level features of Windows Files, Pipes,
Processes, Threads, Services, Event Log and so forth.
Python and similar languages really not suitable for
device-driver type development, and other more systemlike Systems Programming!

OReilly

Windows System Prog

Slide 7

Why not just use Python?

Python

has excellent native support for files, processes,

threads etc.
These features are typically limited to those defined by ANSI C.
Many advanced Windows features are not exposed using these interfaces.
Standard implementation of some of the standard library functions leaves a
little to be desired in a Windows environment.

OReilly

Windows System Prog

Slide 8

Portable Process Control (1 of 3)

Standard

Python functions all work

Just often not quite how we would like!

os.system()
import os
os.system(notepad C:\\autoexec.bat)

Problems
Creates a new console window when run from a GUI.
Waits for process to terminate.

OReilly

Windows System Prog

Slide 9

Portable Process Control (2 of 3)

os.execv

family

Doesnt search system path, and doesnt parse command lines

os.execv("c:\\Winnt\\notepad.exe", \
("c:\\autoexec.bat",) )

Does clobber your existing process - the call to os.execv()

never returns!

OReilly

Windows System Prog

Slide

Portable Process Control (3 of 3)

os.popen()

>>> file = os.popen("echo Hello")

>>> file.read()
'Hello\012'
Works fine from Windows NT console programs, but fails miserably from a
GUI!
win32pipe module in the Win32 extensions provides a working replacement.

OReilly

Windows System Prog

Slide

Better Process Control (1 of )

win32api

module provides some high-level, Windows specific

functions.
win32api.WinExec()
Very similar to os.system(), but overcomes limitations.
>>> import win32api
>>> win32api.WinExec("notepad")
Optional parameter allows you to specify the Windows initial state (eg,
minimized)

OReilly

Windows System Prog

Slide

Better Process Control (2 of 2)

win32api.ShellExecute()

Typically opens documents - eg, execute foo.doc, and (typically) Word will
open.
Finer control over the new process.
Can also execute arbitrary executables - not limited to documents.
For example, to print a specific document:
win32api.ShellExecute(0, "print", \
"MyDocument.doc", None, "", 1)

OReilly

Windows System Prog

Slide

Ultimate Process Control (1 of 2)

win32process

module exposes the low level Win32 API.

Full support for CreateProcess, CreateProcessAsUser,
CreateThread etc.
Full support for Windows Handles
Files can be passed as stdin/out/err
Process and thread handles are waitable using the win32event module.

OReilly

Windows System Prog

Slide

Ultimate Process Control (2 of 2)

Able

to set thread and process priority and affinity

levels
Very handy for bugs that only appear in multi-processor
machines.

Able

to do all this for both existing and new processes.

Process usage samples included in distribution.

OReilly

Windows System Prog

Slide

Introduction to our Sample

Full

sample Windows NT Service

Provides event log and performance monitor information. Clients connect using
Named Pipes
Less than 75 lines of code for all this functionality.
Still too big to present in one hit
Selected

excerpts included in slides

Full code on CD, and at
http://starship.python.net/crew/mhammond/conferences/ora99

OReilly

Windows System Prog

Slide

Portable Files and Pipes

Python

has excellent built-in file support

Inherits platform stdio support - wont bother discussing them here.

Native

Windows files only useful for highly-advanced features, such as:

Overlapped IO
IO Completion Ports
Named Pipes
NT Security requirements

OReilly

Windows System Prog

Slide

Native Files (1 of 4)
win32file.CreateFile()

used for most file operations

Create and open regular files, memory mapped files, etc.

Takes seven parameters - c.f. open()s two!
Returns a PyHANDLE object
Can

be passed to any Python API wrapper expecting a handle.

Auto-closed when last reference removed.

OReilly

Windows System Prog

Slide

Native Files (2 of 4)
Overlapped

IO for asynchronous operations

File opened for overlapped IO requires a Windows event object.

All IO operations return immediately.
Event object signalled when IO complete
Great

for high-performance servers

Simple to support multiple concurrent file operations per thread.

OReilly

Windows System Prog

Slide

Native Files (3 of 4)
NT

Completion ports for even better asynchronous control

Windows manages associating the completed IO operation with a

connection
More complex to use.
Often requires a state-machine implementation.
Offers excellent performance - Microsofts recommended architecture
for scalable, high-performance servers.

OReilly

Windows System Prog

Slide

Native Files (4 of 4)
Full

support for NT Security

Default security can be used by passing None

Many real-world applications require explicit security configuration
Full

support for Windows Impersonation

Processes can automatically impersonate the remote pipe client.

Impersonate any user given their password.

OReilly

Windows System Prog

Slide

File Sample (1 of 2)
Overlapped

IO is used

self.overlapped = \
pywintypes.OVERLAPPED()
# create the event to be used.
self.overlapped.hEvent = \
win32event.CreateEvent(None,0,0,None)

Special security for pipes is needed for services

sa = win32security.SECURITY_ATTRIBUTES()
# Allow full access!
sa.SetSecurityDescriptorDacl ( 1, None, 0 )

OReilly

Windows System Prog

Slide

File Sample (2 of 2)
Create

pipe and connect to client

pipeHandle

= \
win32pipe.CreateNamedPipe(pipeName,
openMode,
pipeMode,
win32pipe.PIPE_UNLIMITED_INSTANCES,
0, 0, 6000, # 6 second timeout.
sa)
...
hr = win32pipe.ConnectNamedPipe(pipeHandle,\
self.overlapped)

OReilly

Windows System Prog

Slide

Windows NT Services
Similar

concept to a Unix daemon

A few special requirements
Must respond to asynchronous commands from NT to (e.g.)
Shutdown
Must be capable of reporting status to NT
NT

has built-in UI for configuring and controlling services.

OReilly

Windows System Prog

Slide

Python controlling Services

Full

exposure of the Windows NT Service Control Manager API

Start, Stop, Pause Services, Install or Remove services, etc

win32serviceutil

module makes it simple

>>> win32serviceutil.StopService("Messenger")
(32, 3, 0, 0, 0, 6, 20000)
>>> win32serviceutil.StartService(...)
>>>

OReilly

Windows System Prog

Slide

Implementing Services
Simple

to implement Services in Python

Smallest service is around 16 lines of code!

Includes

debug support, self-installation, and fully controllable from Windows NT

Few more lines needed to make it something useful :-)

Simply

sub-class Service base-class, and implement control features

Minimum required is StopService

Trivial to implement most controls

OReilly

Windows System Prog

Slide

Windows NT Event Log

Central

place for programs to log information.

Ideal for Services - can not present effective GUIs

Benefits

to programmer

Built in transaction and thread safety, maximum size ability, etc.

Benefits

to Administrator

Central log of messages, and third party tools to help analysis.

OReilly

Windows System Prog

Slide

Python reading the Event Log

Complex

native API from win32evtlog

Simpler interface from win32evtlogutil
Define Feeder function
def DumpRecord(record):
print Got event ID, record.EventID

And feed it!

win32evtlogutil.FeedEventLogRecords( \
DumpRecord)
Got Event ID -2147483645
Got Event ID -2147483645

OReilly

Windows System Prog

Slide

Python writing the Event Log

More

complex than reading - event sources

must be registered.
win32evtlog and win32evtlogutil used here too
Python Service framework also supports
simple event log writing

OReilly

Windows System Prog

Slide

Writing the Event Log Sample

Our

sample uses the Service Framework functions which makes

it trivial
Events are logged by ID, rather than explicit text.
Our sample uses a built-in ID to log a service starting message
import servicemanager
servicemanager.LogMsg(
servicemanager.EVENTLOG_INFORMATION_TYPE,
servicemanager.PYS_SERVICE_STARTED,
(self._svc_name_, ''))

OReilly

Windows System Prog

Slide

NT Performance Monitor
Built-in

NT tool for monitoring an applications performance.

Application must be written to provide this data.
API designed for smallest impact on program supplying
data
Moves the burden to the collecting application.
Not trivial to work with

OReilly

Windows System Prog

Slide

Python reading from PerfMon

Useful

for overcoming limitations in built-in tool

For example, sample the data hourly and log to a longer-term database.
Useful for reading standard information about another process.
Process

ID, Memory usage, etc.

win32pdhutil

module for reading data

Good sample code in the source file

OReilly

Windows System Prog

Slide

Python supplying PerfMon

Complex

installation and setup procedure.

C .h file and custom .ini file necessary at application installation

Supported by the Python Service framework
Quite

trivial to use once running

Simply increment a counter - Performance Monitor handles

conversions to required units.

OReilly

Windows System Prog

Slide

PerfMon Sample
Installation

more than we can cover here

See InitPerfMon() method

Working

with counters trivial

Increment our connections counter each connection

self.counterConnections.Increment()

Perfmon

manages converting to connections-per-second

automatically.

OReilly

Windows System Prog

Slide

Our Sample in Detail (1 of 8)

PipeService2.py

The service implementation.

PipeService2_install.h

PipeService2_install.ini
Required for performance monitor installation
Most services dont need this!
PipeServiceClient.py

Sample client to connect to our service.

OReilly

Windows System Prog

Slide

Our Sample in Detail (2 of 8)

Service

functionality in PipeService class

Base class handles most of the grunt

We simply supply service name and other optional attributes
class PipeService(\
win32serviceutil.ServiceFramework):
_svc_name_ = "PythonPipeService"
_svc_display_name_ = "A sample Python ... "

OReilly

Windows System Prog

Slide

Our Sample in Detail (3 of 8)

We

respond to an NT ServiceStop request by telling NT we are stopping,

and setting a Windows Event

def

SvcStop(self):
# Before we do anything, tell the
# SCM we are starting the stop process.
self.ReportServiceStatus( \
win32service.SERVICE_STOP_PENDING)
# And set my event.
win32event.SetEvent(self.hWaitStop)

OReilly

Windows System Prog

Slide

Our Sample in Detail (4 of 8)

Overlapped

IO means we can wait for either a connection, or our Stop

request
win32pipe.ConnectNamedPipe(pipeHandle,\
self.overlapped)
...
# Wait for either a connection, or
# a service stop request.
waitHandles = self.hWaitStop, \
self.overlapped.hEvent
rc = win32event.WaitForMultipleObjects(\
waitHandles, 0, timeout)

OReilly

Windows System Prog

Slide

Our Sample in Detail (5 of 8)

Install

our service simply by executing the service script

If not for Performance Monitor, the command-line would be simple:

C:\Scripts> PipeService2.py install

But PerfMon needs an extra arg

C:\Scripts> PipeService2.py \
--perfmonini=PipeService2_install.ini install
Installing service PythonPipeService to ...
Service installed

OReilly

Windows System Prog

Slide

Our Sample in Detail (6 of 8)

Once

installed, we can start the service

Start from Control Panel, or using the script itself

C:\Scripts> python.exe PipeService2.py start

And

start a client test session

C:\Scripts> python.exe PipeServiceClient.py \

Hi there
The service sent back:
You sent me:Hi there

OReilly

Windows System Prog

Slide

Our Sample in Detail (7 of 8)

We

will have Event

Log records...

OReilly

Windows System Prog

Slide

Our Sample in Detail (8 of 8)

And

Performance
Monitor Data.

OReilly

Windows System Prog

Slide

Summary
Python

library flexible, rich and portable enough for many tasks

Low-level Windows programming achieved by using
extensions that expose the raw Windows API
This talk should have shown:
An overview of how this programming is done in Python.
How simple it is to do complex Windows tasks.

OReilly

Windows System Prog

Slide

More Information
Python

and Python Documentation

http://www.python.org
http://www.python.org/doc
Python

for Windows Extensions

http://starship.python.net/crew/mhammond
http://www.python.org/windows
Reference Manuals and Samples included with the distributions

OReilly

Windows System Prog

Slide

Thanks for coming

Mark

Hammond

mhammond@skippinet.com.au
http://starship.python.net/crew/mhammond

OReilly

Windows System Prog

Slide