RSS Feed

RSS

Comments RSS

Pyinstaller and cx_Oracle success!

I have had success with producing a single file executable from a Python script that includes cx_Oracle.  Here are the details.  Another post in the PyInstaller Google Groups forum from the past put me onto the right track.  Hopefully this will also help someone struggling to get this working.
OS was Windows 7 Pro 64 bit.
Python was 2.7.8 32 bit.
cx_Oracle-5.13 for 2.7
PyInstaller-2.1
Python script oracle_connect.py, this was my program.
Oracle 11g instant client DLL files, OCI.dll and oraociei11.dll (OCI.dll seems to stay constant from version to version of Oracle, but the other DLL file’s name will change with Oracle versions.)
A properly designed tnsnames.ora file.  This contains the necessary connection information for your Oracle database.
A Pyinstaller-2.1 .spec file to make sure Pyinstaller did everything that was needed to produce the single file .exe.  Pyinstaller-2.1 will not automatically include both DLLs into the .exe without using a .spec file.  It does pick up the OCI.dll file automatically, but not oraociei11.dll.
First, I began by writing my Python script, which imports cx_Oracle, makes the connection to the database, and does some work.  My working directory was C:\Python27, so the script, two DLL files and .ora file were all placed there.  Test the code to make sure that via the interpreter the script interacted with the database correctly.
Second, I ran Pyinstaller-2.1 against the oracle_connect.py script without the -F option (for single file .exe).  This created an oracle_connect.spec file which I used as a template which was edited to include the needed settings.
Third, I edited the oracle_connect.spec file (see below) to specify a single file executable and to include the oraociei11.dll in the compilation  The paths were specific to my installation environment and where I had placed by Python source file and other files, adjust to match your environment.  This oracle_connect.spec file I then placed in my c:\Python27\PyInstaller-2.1 direcory.  Then I executed the following to produce the single file executable.
C:\Python27\PyInstaller-2.1>..\python.exe .\pyinstaller.py .\oracle_connect.spec
The resulting oracle_connect.exe may now be distributed without including loose DLLs.  The only additional file needed is the tnsnames.ora file.  This cannot be compiled into the program and must reside in the same location as the executable program.  It is a configuration file for the communications parameters necessary to make the connection to the Oracle database.  I include a sample of a tnsnames.ora file below.
oracle_connect.spec file contents:
# -*- mode: python -*-
a = Analysis([‘..\\oracle_connect.py’],
pathex=[‘C:\\Python27\\PyInstaller-2.1\\oracle_connect’],
hiddenimports=[],
hookspath=None,
runtime_hooks=None)
pyz = PYZ(a.pure)
exe = EXE(pyz,
a.scripts,
a.binaries  + [(‘oraociei11.dll’,’c:\python27\oraociei11.dll’,’BINARY’)],
a.zipfiles,
a.datas,
name=’oracle_connect.exe’,
debug=False,
strip=None,
upx=False,
console=True )
Sample tnsnames.ora, values in <> are Oracle installation specific, contact your local Oracle admin for the correct values to use.
# tnsnames.ora Network Configuration File
<Oracle instance name> =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = <FQDN or IP>)(PORT = <1521>))
)
(CONNECT_DATA =
(SERVICE_NAME = <Oracle service name>)
)
)

The Watch Folder

For the programmers and systems administrators among us.  Consider the following, and all too common, programming task. Events from one process need to trigger actions by another.  Now, the impulse may be to try to impliment a direct method of communications between the two process via networking sockets, low level OS signals, etc.  While this may be good to consider or even use at times, there may be a far more simple and less problematic method that is almost always worth considering – the Watch Folder.    If both processes can have access to a common directory, via network share or direct filesystem mount for instance, then all one has to do is have the first process drop a small file to the locaton and have the other process be watching for it, act in response and then clear it out and wait again for the next time the file appears in order to act again.   The second process can watch for it directly or have some other task be the watcher that then calls the final process, perhaps through a periodic call from a task scheduler such a cron.   There are lots of realitively simple ways of implementing such an arrangement without resorting to more complex networking or OS signaling programming schemes.   The Watch Folder methodology works well in any language but is particularly easy to implement in simple scripting languages that are available as part of the command shell of most any OS, such as BASH in Linux, Bourne or Korn Shell in Unix, or DOS Batch or Powershell script in Windows.

The Commandline Bible program is released!

My previous post concerned a little program I wrote one evening to process a bible text file and produce a database from it using Python.   That evening project took on a life of its own, so I have removed the archive download from that blog post as it is now very obsolete.   I now introduce and release to the public my Commandline Bible program.  It is a capable bible search and study program designed for use from the Linux or Windows commandline.  Both are writen in Python and require the installation of the Python 2x (currently 2.7.3) interpreter.   The program should work fine for any Python 2.5 or later 2x installation.  In both versions, extract the archive to folder.   For Linux, run python kjv_cmdline.py at the commandline.   For Windows, run the bible.bat batch file which ensures that the needed ANSI character support is loaded and then runs the Python interpreter and the program.   Many thanks to my good friend Larry D. Barr for many suggestions and beta testing support.  This is version 1.0.  Enjoy!

Features:

  • high quality KJV Pure Cambridge Edition (PCE) text is used
  • case sensitive single word or words found in series search
  • search can be for whole bible or limitied to Old Testament, New Testmant, or single Book of the Bible
  • verse display by reference
  • display of the Treasury of Scriptural Knowledge (TSK) references for verses
  • a simple ANSI color text interactive command line program not requiring any graphical interface at all
  • SQLite3 database and other associated datafiles made freely available
  • all software distributed via the free MIT lincese and other applicable free licneses
  • very low system resource usage and very fast

http://dl.dropbox.com/u/28301815/kjv_cmdline.zip

http://dl.dropbox.com/u/28301815/kjv_cmdline.md5

http://dl.dropbox.com/u/28301815/kjv_cmdline-win.zip

http://dl.dropbox.com/u/28301815/kjv_cmdline-win.md5

 

Using Python to create a Bible database program

The programming bug bit me this evening and I did something I have wanted to do for a really long time, write my own code to create a useable bible database.   So, I took an indexed bible text available in the public domain, wrote the code to parse it properly and then package the parsed bible into a Sqlite3 database file that can then be querried.  Basically, the heart of any bible search program.  I cannot believe how fast it runs!  That is a whole lot of text, but Python excells in the text processing domain.   The link is to an archive containing the bible text file that gets processed and the Python program.   When run, the database is created from scratch.

The code is Python 2.x style.   You may use this code in anyway you wish. NOTE: the following code is obsolete and has been removed.  Please see this blog post for the fully featured program!   http://wa5pb.freeshell.org/motd/?p=854


http://dl.dropbox.com/u/28301815/kjvbible.tar.gz

The code has good examples of:
1)reading a text file, line by line
2)text parsing and regular expressions
3)error handling
4)Sqlite3 database handling

Python, watermark a PDF

This blog entry shows how to use Python and two third party modules (pyPdf and ReportLab) to watermark a PDF.

#This sample uses two third part modules for Python, 
#pyPdf & ReportLab to achieve creating and placing 
#watermark text at angle on an existing PDF file. 
#This example was produced with Python 2.7 
#See http://pybrary.net/pyPdf for more informaton about pyPdf. 
#See http://www.reportlab.com for more information about ReportLab. 

#Import the needed external modules and functions from pyPdf and reportlab.
from pyPdf import PdfFileWriter, PdfFileReader 
from reportlab.pdfgen import canvas

#Use reportlab to create a PDF that will be used 
#as a watermark on another PDF.
c= canvas.Canvas("watermark.pdf") 
c.setFont("Courier", 60)
#This next setting with make the text of our 
#watermark gray, nice touch for a watermark.
c.setFillGray(0.5,0.5)
#Set up our watermark document. Our watermark 
#will be rotated 45 degrees from the direction 
#of our underlying document.
c.saveState() 
c.translate(500,100) 
c.rotate(45) 
c.drawCentredString(0, 0, "A WATERMARK!") 
c.drawCentredString(0, 300, "A WATERMARK!") 
c.drawCentredString(0, 600, "A WATERMARK!") 
c.restoreState() 
c.save() 

#Read in the PDF that will have the PDF applied to it.
output = PdfFileWriter() 
input1 = PdfFileReader(file("original_pdf.pdf", "rb")) 

#Just to demo this function from pyPdf. 
#If the PDF has a title, this will print it out.
print "title = %s" % (input1.getDocumentInfo().title)

#Open up the orgininal PDF.
page1 = input1.getPage(0)

#Read in the file created above by ReportLab for our watermark.
watermark = PdfFileReader(file("watermark.pdf", "rb"))
#Apply the watermark by merging the two PDF files.
page1.mergePage(watermark.getPage(0))
#Send the resultant PDF to the output stream.
output.addPage(page1)

#Just to demo this function from pyPdf. 
#Return the number of pages in the watermarked PDF.
print "watermarked_pdf.pdf has %s pages." % input1.getNumPages()

#write the output of our new, watermarked PDF.
outputStream = file("watermarked_pdf.pdf", "wb") 
output.write(outputStream) 
outputStream.close()

Python & SQLite3, demonstrating parameters

Python affords a great opportunity for one to learn basic SQL commands and operations by having SQLite3 built into its standard library.   To learn more, here are some links for Python’s implementation of the API and also the SQLite3 website.

http://docs.python.org/library/sqlite3.html

http://www.sqlite.org/index.html

Here is a brief example of  creating a simple SQLite3 database in Python, performing a query against it with parameters.  The use of parameters is demonstrated in 3 ways.

>>> import sqlite3
>>> conn = sqlite3.connect(‘c:\sqlite3\example’)
>>> c = conn.cursor()
>>> c.execute(“””create table contacts(client_id integer primary key, first_name text, last_name text, email text)”””)
>>> c.execute(“””insert into contacts values(NULL, ‘Bob’, ‘Dole’, ‘bob.dole@gmail.com’)”””)
>>> conn.commit()
>>> c.execute(“””insert into contacts values(NULL, ‘Tom’, ‘Cruz’, ‘tom.cruz@xenu.com’)”””)
>>> conn.commit()

First parameters are passed using the ? placeholder.  The values are supplied by a collection, a list in this example.  A tuple could also be used.

>>>params = [‘Tom’,’Bob’]

>>> c.execute(“””select client_id, first_name, last_name, email from contacts where first_name in (?,?)”””, params)
>>> for i in c:
print(i)

Second, a mapping object (dictionary) is used to supply the parameters via its keys.   This allows the parameters to be named.

names = {“name1”: ‘Bob’, “name2”:’Tom’}

>>> c.execute(“””select client_id, first_name, last_name, email from contacts where first_name in (name1,name2)”””, names)
>>> for i in c:
print(i)

Third, we use the built in mapping that stores information about local program variables to supply the parameters.  The locals() function is used to return the mapping information.  This allows the parameters to be simple variables, rather than members of a collection or keys in a mapping.

name1 = ‘Bob’; name2 = ‘Tom’

>>> c.execute(“””select client_id, first_name, last_name, email from contacts where first_name in (name1,name2)”””, locals())
>>> for i in c:
print(i)

All three methods produce the very same result of the query on the database:

(1, ‘Bob’, ‘Dole’, ‘bob.dole@gmail.com’)
(2, ‘Tom’, ‘Cruz’, ‘tom.cruz@xenu.com’)
>>> c.close()

Additionally, remember the parameter needs to be passed to the execute() function by a collection or mapping type variable.   So, if you would like to loop through the database, say by the indices, and are using a simple integer variable in the loop, you can take that and convert it to a list variable on the fly (a list containing one element) and use that.    Like this:

out_list = []
for x in range(1,4):
    y = [x]        #here we take the integer x and use it to create the single element list y for use in c.execute()
    c.execute("""select client_id, first_name, last_name, email from contacts where client_id = :y""", y)
    for i in c:
        out_list.append(i)
print(out_list)

A geek celebration of Easter

def EasterDate(y):

# EASTER DATE CALCULATION FOR YEARS 1583 TO 4099

# y is a 4 digit year 1583 to 4099
# d returns the day of the month of Easter
# m returns the month of Easter

# Easter Sunday is the Sunday following the Paschal Full Moon
# (PFM) date for the year

# This algorithm is an arithmetic interpretation of the 3 step
# Easter Dating Method developed by Ron Mallen 1985, as a vast
# improvement on the method described in the Common Prayer Book

# Because this algorithm is a direct translation of the
# official tables, it can be easily proved to be 100% correct

# This algorithm derives values by sequential inter-dependent
# calculations, so … DO NOT MODIFY THE ORDER OF CALCULATIONS!

# The / operator may be unfamiliar – it means integer division
# for example, 30 / 7 = 4 (the remainder is ignored)

#The % operator is for the Modulus operation

# All variables are integer data types

# It’s free! Please do not modify code or comments!
# ==========================================================
#The following is a translation of the BASIC code for the calculation
#of Easter dates to Python 2x. The comments from that code and
#variable names have been preserved. The preceding comments
#have also been preserved with only slight modification and additions
#that are necessary for properly understanding this Python version
#of the program. I encourage the reader to visit following website
#and compare the original BASIC code with the Python code presented
#here.
#The original documentation and code may be found at:
#http://www.assa.org.au/edm.html
#http://www.assa.org.au/edm.html#Computer
#
#Many thanks to the Astronomical Society of South Australia for
#the information that made this possible.
#http://www.assa.org.au/
#FirstDig, Remain19, temp are intermediate results
#tA, tB, tC, tD, tE are table A to E results

FirstDig = y / 100 #first 2 digits of year
Remain19 = y % 19 #remainder of year / 19

#calculate PFM date
temp = (FirstDig – 15)/2 + 202 -11 * Remain19

case1 = (21,24,25,27,28,29,30,31,32,34,35,38)
case2 = (33,36,37,39,40)

if FirstDig in case1:
temp = temp – 1
elif FirstDig in case2:
temp = temp – 2

temp = temp % 30

tA = temp + 21
if temp == 29:
tA = tA -1
if temp == 28 and Remain19 > 10:
tA = tA -1

tB = (tA -19) % 7

tC = (40 – FirstDig) % 4
if tC == 3:
tC = tC + 1
if tC > 1:
tC = tC + 1

temp = y % 100
tD = (temp + temp/4) % 7

tE = ((20 – tB – tC – tD) % 7) + 1
d = tA + tE

if d > 31:
d = d – 31
m = 4
else:
m = 3

return (m,d)

#A simple program using the EasterDate funtcion
year = int(raw_input(“Find the date of Easter for which year(1582-4099)? “))
month_day = EasterDate(year)
months = (‘January’, ‘February’, ‘March’, ‘April’, ‘May’, ‘June’, ‘July’,
‘August’, ‘September’, ‘October’, ‘November’, ‘December’)
print “For the year {0}, Easter will fall on {1} {2}”.format(year, months[month_day[0]-1],month_day[1])

 

a different kind of ham homebrewing

This little project just took on a life of its own this afternoon.  What started out as an experiment in writing a single program to read output data from my radio via the RS232 serial interface has turned into a full blown radio remote control program.

The current state of the program will let me control my rig from a remote PC across the local network at my QTH.

There are 4 separate programs involved in this.  Much of the code from my previous blog post comprises the heart of the system.

Here is how it all works. On the PC that is connected to the radio (we will call this the PC-Radio system) resides two programs – K2client.py and K2server.py.   On the remote PC across the network (we will call this the PC-operator system) resides two other programs – client.py and server.py.   On the PC-Radio system, the programs each have to communicate with both the network and the radio.   The K2client.py program takes any output data from the radio and sends it out to the network to be received as feedback from the radio by the server.py program on the PC-operator system. The K2server.py program receives any data sent to it from across the network by the client.py program on the PC-operator system and passes these as commands to the radio.  The radio takes the command, gives feedback, and cycle begins again.   As a diagram, it looks like this.

server.py[PC-operator system]client.py —————->K2server.py [PC-Radio system]K2client.py —>
^                   (operator)                    (network)                             (radio)                        |
|                                                                                                                                v
<——————–<———————————<—————————–<——————————-<——————–<—————————-

It was not as that this was all entirely necessary.  I could have downloaded software already designed to do this, and have used such in the past with my computers and radio.   However, for me, it is the difference between buying a radio and building a radio.  You can accomplish the very same thing in the end, e.g. communication.  Both ways are OK.  However, there are added benefits from building equipment such as the opportunity to learn, pride in the work one has done, the sense of accomplishment, etc.   The same is true for this type of homebrewing, the only difference is that it is homebrewed software rather than hardware.  The Python source code for the project is available.  This code is very minimalist and intended only to demonstrate the principles involved.  Please feel free to use, adapt, or modify it in any way you may wish.

NOTE:  The code from my previous blog post contains comments that explain how to take this Linux implementation of RS232 communications and adapt it for use on a Windows system.  Also note that there are hard coded IP addresses in the code that would need to be changed for implementation elsewhere.  Aside from those differences, this code should be fully cross-platform and usable anywhere as it pertains to the parts dealing with the serial and network communications.  The control codes and settings I am sending to my radio may not be applicable to your own, so please see the users or programming manual for your rig for appropriate codes, settings and methods for communicating and controlling it.   Additionally, the code found in this project that implements the network socket clients and servers are adaptations of code found at evolt.org.

Python Server to Client socket communications

I found a terrific little tutorial on the subject of programming network socket Server/Client communications in Python. Very clear and concise! The author did a nice job with this.

Socket Programming in Python @ evolt.org

Serial Communications using Python

#k2test1.py  by Bill Allen – WA5PB
#This is an experiment in communicating with an Amateur Radio transceiver
#via Python through RS232 serial communications.  Similar code could be used
#to communicate with any device supporting serial communications.

#This was written using Python 2.6.5, but I like using the new
#Python 3.x print function so we pull it in for use here.
from __future__ import print_function

#We will be using the PySerial library to provide RS232 serial communications.
#PySerial is a third party library module that can be found here:
#http://pyserial.sourceforge.net/
import serial

#Configure the serial port to communicate with the Elecraft K2 xcvr.
#I am coding this on a Linux system, so the device string /dev/ttyS0
#works for the first serial port.  Your system my use a different serial
#port device.  For a Windows system, try COM1 for the first serial port.
ser = serial.Serial(‘/dev/ttyS0’, 4800, timeout=1)

#Sanity check for the K2 AutoInfo mode, turn it off.
ser.write(“AI0;”)

#Now set the K2 AutoInfo mode to mode 1 and get some initial data.
ser.write(“AI1;”)

#Read up to 150 bytes of returned data from the K2 and print it out.
K2_data = ser.read(150)
print(K2_data)

#Now set the K2 AutoInfo mode to return the most data while it is being polled.
#Ctrl-C will be used to end the program.  End the program gracefully by trapping
#the Ctrl-C, turning off the K2 AutoInfo mode and closing the serial port prior
#to exiting.
ser.write(“AI3;”)
while True:
    try:
        K2_data = ser.read(150)
        if K2_data != “”:
            print(K2_data)
    except KeyboardInterrupt:
        print(“\nOK, exiting…”)
        print(“Reseting K2 AutoInfo mode OFF”)
        ser.write(“AI0;”)
        print(“Closing Serial Port”)
        ser.close()
        raise SystemExit(0)

PyFITS Cheat Sheet

PyFITS is a Python module to facilitate the reading, editing and creation of Flexible Image Transfer System (FITS) files.  FITS is a common data interchange medium in the astronomical community.

PyFITS Cheat Sheet, working with the FITS file Header Data Unit(HDU)

extracted from The PyFITS Handbook
http://stsdas.stsci.edu/download/wikidocs/The_PyFITS_Handbook.pdf

see also:  A Primer on the FITS Data Format

#load the pyfits module into Python
import pyfits

#initialize a fits file object
hdulist = pyfits.open(‘input.fits’)

#when finished close the file object
hdulist.close()

#get info from the fits file
hdulist.info()

#get the value of a particular keyword in the HDU
hdulist[0].header[’targname’]

#get the value of a particular keyword index in the HDU
hdulist[0].header[27]

#assign new values to keywords or indexes in the HDU
prihdr = hdulist[0].header
prihdr[’targname’] = ’NGC121-a’
prihdr[27] = 99

#view the entire HDU contents
prihdr = hdulist[0].header
print(prihdr)

#update (add) the HDU with a new keyword and value
prihdr.update(’observer’, ’Edwin Hubble’)

#add history or comment records to HDU
prihdr.add_history(’I updated this file 2/26/09’)
prihdr.add_comment(’Edwin Hubble really knew his stuff’)

#access the HDU as Cards, display first three cards of HDU
print prihdr.ascardlist()[:3]
SIMPLE =    T / file does conform to FITS standard
BITPIX =    16 / number of bits per data pixel
NAXIS =     0 / number of data axes

#list all the keywords of the HDU
prihdr.ascardlist().keys()
[’SIMPLE’, ’BITPIX’, ’NAXIS’, …]

#write out the changes made to the in memory fits object
hdulist.writeto(’newimage.fits’)

#saving changes to fits file using update mode
f = pyfits.open(’original.fits’, mode=’update’)
… # making changes in data and/or header
f.flush() # changes are written back to original.fits

Python Dictionaries, a native Hash Table data type

All programming languages make data structures available for data storage.   These can be very simple and small as registers and memory byte, word, or multi-word variables as in various Assembly languages.  They can also be potentially large and complex as memory pointer linked lists or multidimensional arrays that are available in many high level languages.   Python provides for many types of data structures as well, the best known being its List type.   While this is definitely the workhorse of the language, Python also provides something even more flexible and powerful – Dictionaries.   In most programming languages, a dictionary would also be known as a “hash table”.   A hash table differs from an array or list in that it allows for the access of data by named indices that are defined by the programmer, rather than by integer indices only.   In most programming languages, these hash tables are cobbled to together by the programmer and often require a lot of hand coding and handling of pointer references.   Python makes dictionaries available as a base data type and one only has to create the dictionary variable and begin assigning data to it.  That’s it!  No additional coding is required to create the dictionary or maintain it aside form what ever operations are desired to add, delete, or otherwise maintain its contents – and native functions are provided such as, but not limited to, ‘del’, ‘keys’, ‘values’,  ‘items’, and ‘pop’.

Here is an example.  It is a nested dictionary, or dictionary of dictionaries, that contains a variety of data.

>>> my_dictionary = {‘3262812’: {‘SAP_state’: ‘False’, ‘REV’: ‘B’, ‘SPECS’: [‘Q00021’, ‘Q00024’]}, ‘3231023’: {‘SAP_state’: ‘True’, ‘REV’: ‘C’, ‘SPECS’: [‘Q00022’, ‘Q00026’, ‘Q00027’, ‘E50002’]}, ‘3257313’: {‘SAP_state’: ‘True’, ‘REV’: ‘O’, ‘SPECS’: [‘Q00021’, ‘Q00020’, ‘Q00025’, ‘M25070’, ‘Q50151’, ‘Q50153’]}}

First we create a dictionary called my_dictionary.   Notice that dictionaries use {} for their enclosing brackets, not [] used by lists.   However, dictionaries DO use [] for later reference and access to the dictionary as will be seen below.   For example, the first entry in our dictionary shown here is ‘3257313’: {‘SAP_state’: ‘True’, ‘REV’: ‘O’, ‘SPECS’: [‘Q00021’, ‘Q00020’, ‘Q00025’, ‘M25070’, ‘Q50151’, ‘Q50153’]}.   The components of this dictionary entry is a part number – ‘3257313’ which is a KEY to this dictionary and another dictionary with keys ‘SAP_state’, ‘REV’, and ‘SPECS’.  ‘SAP_state’ gives access to a string variable, ‘REV’ to a string variable, and ‘SPECS’ to a list variable that contains strings.

Access to data of the dictionary and had by one or more keys as shown.

>>> my_dictionary[‘3257313’]
{‘SAP_state’: ‘True’, ‘REV’: ‘O’, ‘SPECS’: [‘Q00021’, ‘Q00020’, ‘Q00025’, ‘M25070’, ‘Q50151’, ‘Q50153’]}
In this case, we accessed a main key, which yield all the dictionary contents for that key.

>>> my_dictionary[‘3257313’][‘REV’]
‘O’
Here we access via the a main key and subkey ‘REV’.

>>> my_dictionary[‘3257313’][‘SAP_state’]
‘True’
Here we access via the a main key and subkey ‘SAP_state’.

>>> my_dictionary[‘3257313’][‘SPECS’]
[‘Q00021’, ‘Q00020’, ‘Q00025’, ‘M25070’, ‘Q50151’, ‘Q50153’]
Here we access via the a main key and subkey ‘SPECS’

>>> my_dictionary[‘3257313’][‘SPECS’][3]
‘M25070’
Here we go three levels deep into the dictionary and access via the main key ‘3257313’, subkey ‘SPECS’ and then the index [3] of the list found in ‘SPECS’.

Here are examples of what is returned when the ‘keys’, ‘values’, and ‘items’ methods of the dictionary are used.

>>> my_dictionary.keys()
dict_keys([‘3257313’, ‘3231023’, ‘3262812’])

>>> my_dictionary.values()
dict_values([{‘SAP_state’: ‘True’, ‘REV’: ‘O’, ‘SPECS’: [‘Q00021’, ‘Q00020’, ‘Q00025’, ‘M25070’, ‘Q50151’, ‘Q50153’]}, {‘SAP_state’: ‘True’, ‘REV’: ‘C’, ‘SPECS’: [‘Q00022’, ‘Q00026’, ‘Q00027’, ‘E50002’]}, {‘SAP_state’: ‘False’, ‘REV’: ‘B’, ‘SPECS’: [‘Q00021’, ‘Q00024’]}])

>>> my_dictionary.items()
dict_items([(‘3257313’, {‘SAP_state’: ‘True’, ‘REV’: ‘O’, ‘SPECS’: [‘Q00021’, ‘Q00020’, ‘Q00025’, ‘M25070’, ‘Q50151’, ‘Q50153’]}), (‘3231023’, {‘SAP_state’: ‘True’, ‘REV’: ‘C’, ‘SPECS’: [‘Q00022’, ‘Q00026’, ‘Q00027’, ‘E50002’]}), (‘3262812’, {‘SAP_state’: ‘False’, ‘REV’: ‘B’, ‘SPECS’: [‘Q00021’, ‘Q00024’]})])

Another dictionary function that should be mentioned is ‘del’.  It is used as a procedural function to delete entries from the dictionary.  For instance, if the dictionary looks like this:
my_dictionary
{‘3257313’: {‘SAP_state’: ‘True’, ‘REV’: ‘O’, ‘SPECS’: [‘Q00021’, ‘Q00020’, ‘Q00025’, ‘M25070’, ‘Q50151’, ‘Q50153’]}, ‘3231023’: {‘SAP_state’: ‘True’, ‘REV’: ‘C’, ‘SPECS’: [‘Q00022’, ‘Q00026’, ‘Q00027’, ‘E50002’]}, ‘3262812’: {‘SAP_state’: ‘False’, ‘REV’: ‘B’, ‘SPECS’: [‘Q00021’, ‘Q00024’]}}

Then del my_dictionary[‘3257313’] would delete the complete entry in the dictionary referenced by the key ‘3257313’, giving the following result:
>>> del my_dictionary[‘3257313’]
>>> my_dictionary
{‘3231023’: {‘SAP_state’: ‘True’, ‘REV’: ‘C’, ‘SPECS’: [‘Q00022’, ‘Q00026’, ‘Q00027’, ‘E50002’]}, ‘3262812’: {‘SAP_state’: ‘False’, ‘REV’: ‘B’, ‘SPECS’: [‘Q00021’, ‘Q00024’]}}

The same entry deletion operation could also be achieved by using the ‘pop’ method.   The difference is that pop will allow the removed items associated with the key to be assigned to another variable.
Example:
>>> my_dictionary
{‘3257313’: {‘SAP_state’: ‘True’, ‘REV’: ‘O’, ‘SPECS’: [‘Q00021’, ‘Q00020’, ‘Q00025’, ‘M25070’, ‘Q50151’, ‘Q50153’]}, ‘3231023’: {‘SAP_state’: ‘True’, ‘REV’: ‘C’, ‘SPECS’: [‘Q00022’, ‘Q00026’, ‘Q00027’, ‘E50002’]}, ‘3262812’: {‘SAP_state’: ‘False’, ‘REV’: ‘B’, ‘SPECS’: [‘Q00021’, ‘Q00024’]}}
>>> removed_item = my_dictionary.pop(‘3257313’)
>>> removed_item
{‘SAP_state’: ‘True’, ‘REV’: ‘O’, ‘SPECS’: [‘Q00021’, ‘Q00020’, ‘Q00025’, ‘M25070’, ‘Q50151’, ‘Q50153’]}
>>> my_dictionary
{‘3231023’: {‘SAP_state’: ‘True’, ‘REV’: ‘C’, ‘SPECS’: [‘Q00022’, ‘Q00026’, ‘Q00027’, ‘E50002’]}, ‘3262812’: {‘SAP_state’: ‘False’, ‘REV’: ‘B’, ‘SPECS’: [‘Q00021’, ‘Q00024’]}}

Dictionaries are highly flexible and very easily used hash tables in Python.  The advantage of Python dictionaries is direct access to data via named references without having to perform searches via a built in data type.   Just provide a valid key value and Python does the rest.

Hang 10!

As readers of the blog will know, I am a real Python programming enthusiast.  It is gratifying to discover that I am apparently riding quite a large wave of growth in popularity and usage of Python.   This article from Dr. Dobbs Journal discusses this and other interesting programming trends revealed by the Tiobe Programming Community Index.   The Rise and Fall of Languages in 2010

ASCII Mandelbrot Fractal – as a Python CGI

Here is an example of a Python CGI that I have running on my SDF.org website.  It generates a ASCII Mandelbrot Fractal and allows for the user to change and zoom in the field of view.  Granted, it is very low resolution being ASCII, but the main purpose was to demonstrate the use of Python for CGI purposes on SDF.org.   (FOR FURTHER EXPLORATION):   Since the SDF.org server Python package (v 2.6.5) has the PIL (Python Image Library) available, I can see no reason that true binary images of Mandelbrot fractals might not be produced and displayed on as pictures on the web page.  UPDATE:  As suspected, creating true graphical Mandelbrot fractals did in fact work out well.   Here is a link to a Python CGI that uses PIL to create the images.  Graphical Mandelbrot CGI

Ubuntu+XAMPP+Python part 1

Do you have Ubuntu running on a system at home or work?  Have you ever wanted to run your own local web server or at least have one available for development work?   Have you ever wanted to craft your own web CGI scripts?  If you answer yes to these, then here is a serials of tutorials on how to get an Apache web server running quickly and easily on your Ubuntu system and how use Python to craft CGI scripts to run on it.

First, let’s get an Apache web server up and running.   Since this is targeted at people running Ubuntu desktop, rather than Ubuntu server, the very simplest way to install Apache is to do it via XAMPP.   XAMPP is a bundled, self-contained Apache web server distribution provided by Apache Friends that is complete and robust.  It includes  features here as listed from the website:

The distribution for Linux systems (tested for SuSE, RedHat, Mandrake and Debian) contains: Apache, MySQL, PHP & PEAR, Perl, ProFTPD, phpMyAdmin, OpenSSL, GD, Freetype2, libjpeg, libpng, gdbm, zlib, expat, Sablotron, libxml, Ming, Webalizer, pdf class, ncurses, mod_perl, FreeTDS, gettext, mcrypt, mhash, eAccelerator, SQLite and IMAP C-Client.

The XAMPP distribution is designed simply to be downloaded, unpacked, and run from a self-contained directory.   It does not modify your Ubuntu system at all and can be removed simply by deleting the folder that contains the installation.    Now, follow this link to the Apache Friends website and follow the directions for downloading and installing the XAMPP distribution, after reading the notes I provide here.  You will notice there are also instructions on securing the installation.  This is particularly important if you are using this for anything other than development purposes running on your personal system.  If this will be exposed to others, or the world, be sure to follow the instructions on making XAMPP secure.  Note:  Since this is being targeted for Ubuntu do not use the su command as stated in the XAMPP installations instructions.  Instead, you should install using this single line.

sudo  tar xvfz xampp-linux-1.7.3a.tar.gz -C /opt

XAMPP will be installed in the /opt/lampp folder.   To start it up, go to a shell prompt and type the following.  Again, sudo is required.

sudo /opt/lampp/lampp

Now, open you browser and give http://localhost as the address.  You will be presented with the default XAMPP webpage that is being served out by your local Apache web server.  I encourage you to play around with XAMPP for a while and explore your installation via the default page, then return here for the next installment of this series where we will look further at developing CGI scripts in Python.

Python – the new BASIC

I recently read a good blog on the subject of the Python programming language being the new BASIC, and that being a good thing. I very much agree, and it IS a good thing. I got my start in computing and programming just as the age of the home computer was coming to be, i.e. days of the Sinclair, Atari, TRS 80 type home systems. BASIC was the mainstay on these systems and usually came bundled in the system. In those days, home computer users were not looking to just surf the web (which did not exist yet as we know it) or other entirely enduser type activities that are so common today. Back in the day, late 70’s & early 80’s, home computer users were most difinately hobbyists and enthusiasts. Most folks were looking to discover just what they could do with this computer technology now that it was feasible to have it in the home without spending too much of a fortune. Part of this experience for most was learning the version of the BASIC programming language that was bundled, or built into, their computer. There was not much software available back then and the folks were willing and eager to write their own. I’ll never forget the very first computer that our family purchased and on which I first hacked out in BASIC a program. I was a Radio Shack TRS-80 Pocket Computer. It had a HUGE 8Kbytes of RAM! It plugged into and interface that let you print to adding machine roll paper and save programs to an external cassett tape. I learned BASIC and so did my Dad. Those were the days! BASIC was the gateway into programming because it had two fundamental features – it was simple and highly accessible. It came with nearly ALL home computing systems. BASIC spread like wildfire. Today, there is a great need for the same thing. A programming language that is both simple and accessible. While I have mastered several programming languages over the years, the only one that harkens me back to the good old days of BASIC is Python. This is a GOOD thing! Python is both totally free and available for most any computer system and it is simple to learn. It is interpreted and highly portable, just like BASIC was, yet is powerful. Any newbie programmer can pick it up and get it to spit out “Hello World” in just one simple line of code and then within mere days (maybe even hours) be creating useful Object Orientated code using the most modern programming paradigms (usually with even realizing it!). Also, it is not some toy programming language. There is a huge body of high end programs written in it which is growing everyday. Python is the NEW BASIC, without the design shortcomings of BASIC. It is what we have needed.

Doug Hellmann – Code Interstices

I have come across a terrific Python resource in a blog. The blog Doug Hellmann – Code Interstices features a PyMOTW, Python Module Of The Week, that covers much of the Python Standard Library with discussion and examples. The discussion is cogent and the examples clear. Mr. Hellmann has put together an index of all his PyMOTW articles that in total is a very good Python reference. I highly recommend this resource and commend the author for sharing with the community.

Also, Doug Hellmann’s main website.

Python – being exceptional!

One of the key features of Python that often gets overlooked is its ability to deal with exceptions, which is any time a piece of program code could throw a system or runtime error and end abnormally.

The construction that Python provides to handle these situations is the try:/except: combination. The idea is that you have a piece of code that has a high likelihood of creating an error, also call “throwing an exception”. This portion of the code is put in the try: section. If the first piece of code executes normally, without error, the program continues normally. If it traps an exception, then the code in the except: section is executed. In this way, the program continues one way or another in the event of the expected error. This finds its use very often in file handling code where a file open request might fail.

Here is an example. Also, this demonstrates how to do some work in SQLite3 (might as well get two useful examples for the price of one, right?). In this case it is not a file error that is being watched for by the exception handling code, but instead a runtime error that would be created by trying to add a table to a database if the table already exists.

#Create anSQLite3 database.
connection = sqlite3.connect(‘parts.db’) #If the database does not already exist, it is created.
cursor = connection.cursor() #Create an instance of the database cursor.

#Try to create the table “myparts” in the parts.db database. If the table already exists, print an appropriate error message rather than crashing the program with an error and continue with the program.

try:
cursor.execute(‘CREATE TABLE myparts(id INTEGER PRIMARY KEY, part_number VARCHAR(16), part_desc VARCHAR(50), part_type VARCHAR(12))’)
except:
print(“DB already exists”)

HOWTO-Build a searchable dictionary from a .csv using Python

##Building a searchable dictionary from a .csv using Python
##by Bill Allen    10/20/2010

##Sample contents of test.csv, the first row was a header row in the file
##    Material,Material description,Dv,Matl Group
##    256213,IRON KIT STEWART & STEVENSON TRIPLEX,FL,Z-OTF989
##    256214,IRON KIT STEWART & STEVENSON QUINTAPLEX,FL,Z-OTF989

>>> import csv                        #import the csv module for handling of Comma Separated Value files
>>> g = {}                          #create an empty dictionary
>>> reader = csv.reader(open(“test.csv”, ‘r’))         #for the csv module, this reader object is much like a database cursor object
>>> reader_list = list(reader)                    #create a usable list from the reader object (the entire test.csv file contents)

#As read in from the file, each list entry is a list of 4 strings.
#Each could be thought of as a record consisting of:
#[part_number, description, product_line, product_type]
>>> print(reader_list[1])
[‘256213’, ‘IRON KIT STEWART & STEVENSON TRIPLEX’, ‘FL’, ‘Z-OTF989’]
>>> print(reader_list[2])
[‘256214’, ‘IRON KIT STEWART & STEVENSON QUINTAPLEX’, ‘FL’, ‘Z-OTF989’]
>>>

#Now we start building our dictionary.
#In this example, we want the part_number, description, and product_type.
#For each entry in the dictionary the part_number is the key value ((THINK HASH TABLES!)).
#The description and the product_type will be stored under each key as a tuple so each part
#of the dictionary entry is still independently accessible.
>>> g[reader_list[1][0]] = (reader_list[1][1], reader_list[1][3])
>>> g[reader_list[2][0]] = (reader_list[2][1], reader_list[2][3])

#Here is our resulting dictionary.
>>> print(g)
{‘256213’: (‘IRON KIT STEWART & STEVENSON TRIPLEX’, ‘Z-OTF989’), ‘256214’: (‘IRON KIT STEWART & STEVENSON QUINTAPLEX’, ‘Z-OTF989’)}

#Print out individual dictionary entries by part_number.
>>> print(g[‘256213’])
(‘IRON KIT STEWART & STEVENSON TRIPLEX’, ‘Z-OTF989’)
>>> print(g[‘256214’])
(‘IRON KIT STEWART & STEVENSON QUINTAPLEX’, ‘Z-OTF989’)
>>>

#Print out description and product_type information independently by part_number.
#All the data that was stored in the dictionary is fully accessible!
>>> print(g[‘256214’][0])
IRON KIT STEWART & STEVENSON QUINTAPLEX
>>> print(g[‘256214’][1])
Z-OTF989
>>>

#Now we can test search the dictionary for part_numbers.
#Just some simple Boolean tests here to demonstrate.
>>> ‘256214’ in g
True
>>> ‘256213’ in g
True
>>> ‘256216’ in g
False
>>>

#Now for a sample program.
#We will build the dictionary and then print out the first 9 entries.

import csv
parts_dict = {}
reader = csv.reader(open(“test.csv”, ‘r’))
reader_list = list(reader)

#Build the dictionary.
for i in range(1,len(reader_list)):
     parts_dict[reader_list[i][0]] = (reader_list[i][1], reader_list[1][3])

#Print out the first 9 entries with little bit of formatting thrown in.
for j in range(1,10):
     print(“key>”,reader_list[j][0],” – data>”,parts_dict[str(reader_list[j][0])], sep=”)

>>>>
key>256213 – data>(‘IRON KIT STEWART & STEVENSON TRIPLEX’, ‘Z-OTF989’)
key>256214 – data>(‘IRON KIT STEWART & STEVENSON QUINTAPLEX’, ‘Z-OTF989’)
key>256224 – data>(‘IRON KIT STEWART & STEVENSON QUINT PUMP’, ‘Z-OTF989’)
key>3100695 – data>(‘+A RING 2.00 LS20 BRS’, ‘Z-OTF989’)
key>3100720 – data>(‘LF020 LS20 UNIRAW BENT’, ‘Z-OTF989’)
key>3100784 – data>(‘JP060 OLPO SIR F .625 BALL’, ‘Z-OTF989’)
key>3100789 – data>(‘JP060 OLPO SIR M .625 BALL’, ‘Z-OTF989’)
key>3100791 – data>(‘JP100 OLPO SIR M .625 BALL’, ‘Z-OTF989’)
key>3100849 – data>(‘JP120 OLPO SIR F .625 BALL’, ‘Z-OTF989’)

Python – very graphic! cont.

I must say, this has been a great experience trying out some graphical programming in Python. I looked at two different graphics packages for this exercise, Tk and PyGame. While Tk is very good for many graphical interfaces, particularly GUI, it did not provide what I needed most – the ability to do pixel by pixel plotting. Tk, does provide a “way” of doing it using the PhotoImage class, but it was rather cumbersome and felt like a workaround. So, I explored PyGame. It is a mature third party Python library specifically designed for graphical game design. It had exactly what I needed and was very straight forward. I simply wanted to create a display screen area and plot colored pixels to it corresponding to Mandelbrot Set. PyGame gave me exactly what I needed. Here is a series of pictures of the Mandelbrot Set. First, the full set. Second, an image zooming in on a region in the set. Third, zooming in even further. The fascinating thing about the Mandelbrot Set, and all fractal objects, is the the more you zoom in, the more detail you see – not less.

The Mandelbrot Set

Zooming in on the Mandelbrot set #1

Zooming in on the Mandelbrot Set #2