Python Experience Summary

Teaching myself a new language has been a wonderful learning experience, and I am grateful that I have been compelled towards this endeavor during my final semester at John Carroll. I do believe that keeping up to date and continuing learning is critical especially in the fields of software development and computer science. Writing this blog has been a great reminder of that and an inspiration that independent learning outside of a structured classroom environment is very powerful.

This project has certainly had its ups and downs; some days, Python coding and syntax felt like a breeze, but other days not so much. There have been times in this project where I would have wanted to go to the professor to get a quick answer to my questions, but struggling through my problems on my own provided a greater reward in the end. Sometimes I'd be on the wrong track, but I would still be learning through that, and that is why this exercise has been so valuable. There are a crazy amount of resources out there for Python, and there is so much to learn. And then, even if you learn something one way, there may be ten other different ways to do the same thing!

Learning Python has provided me with a new strong point for my resume, as well as a skill that is applicable to my future career. Data Scientists at Progressive Insurance, my future employer, use Python as the language for just about all of their programs. My having this skill will be an invaluable tool as I work my way through the company. I hope that in the future I will continue to learn more in this language and have the opportunity to apply my knowledge. Even if I don't, this experience has taught me how to learn independently and continue my education beyond the bounds of a classroom, and I am all the more excited about what opportunities await for me in the future!

Sending Emails with Python

After a couple hours of struggling to find a template that worked to send a simple email, after 5 versions attempting the code, and after much browsing through Python tutorials that claimed to have the answer, I finally found code that worked! Here is the basic structure of Python code that works (at least for me) to send an email:



The code is pretty simple. I start by importing the Python library smtplib, which allows us to send emails. Then I define constants such as email credentials as well as the message I wish to display, which I will use later. Next, I connect to the server, inputing the proper IP address and port for the gmail server from which I will be sending emails. Note that I created a new gmail account for the purpose of this project and I hid the password in the snip. The line that reads "server.startttls()" is a security function that protects my password when the code is running. The last three lines of code input the login credentials to the server, send the email, and quit the server.

Now, to put this together with the rest of our code.

First, I called the sendEmail method from the main method, inputting all the claim information for the claim whose email is to be sent:



Here is the method to send the email:


And the formatted email that the program successfully sends:



I had a lot of trouble getting a basic email to send. Like I said, I went through at least 5 different sources (and several hours) to find one that had code that worked for me. Even then, I had to make some adjustments for the program to fit my needs. A lot of the confusion for me came from the following bit of code:



What this code does is assign the From and To addresses and the Subject into the appropriate fields for the recipient of the email to see. Doing this, allows the expanded information to display to show the details of the email (sender, recipient, subject):




To test that the program works, to send more than one email, I created a new claim in the csv file with valid information to send to another one of my email accounts. The email did successfully send!

If you have any questions, please do not hesitate to ask in the comments!


Sources:
https://stackoverflow.com/questions/399129/failing-to-send-email-with-the-python-example
http://naelshiab.com/tutorial-send-email-python/

Data Verification

Now that we have uploaded our file, we need to verify that the data we have is valid. To check that the information for each claim is complete, we verify the following:

• That f_name, l_name, and email are not null
• That claim_num is a 5 digit number
• That phone_num is in the format ###-###-####

For each claim that contains errors, we will print out an error indicating what information is missing or invalid. If the data is all valid for a claim, we will state that the data is valid.

Here is the data we are reading from:



Here's the code:

And here is the output:

Notes on the code:
Before going on to read the rest of this post, I encourage you to read through the code provided above. Once you have done that, here are some observations:

• Notice that this code is written using methods. We have a main() method where the program starts, and one additional method called check_phone_validity(). I wrote this method, because the checking of a valid phone number requires more code that a simple conditional statement. I wrote all the other checks in the main() method because they were simple and short enough to handle there. I could have written methods for each check case, but it was not necessary and is really a matter of preference. Note that this code could also have been written without using any methods at all. This is a great example of Python's flexibility and ease of use!
• Since f_name, l_name, and email are strings, to check if they are null, we compare them to "". If we were working with integers, we would compare them to None. This is a distinction I came across and learned about as I was writing the code.
• We use the boolean variable 'clean' to mark if the row of data being checked has any deficiencies, and if it does, clean is set to false, otherwise, the claim is reported to have valid data. In my next blog post, I will add a line at the very end of the method, under the last conditional, so that if clean is True, the program calls a method to send a formatted email to the claimant with the information for that claim. 
• The check_phone_validity() method first checks that the input has 12 characters. Then it goes on to make sure that in positions 3 and 7 in the string, there must be a dash, in keeping with the format requirements. We return whether an invalid phone number affects the validity of the data.

As always, feel free to comment or ask questions in the comments section below!

Sources:

https://stackoverflow.com/questions/15258708/python-trying-to-check-for-a-valid-phone-number
https://www.pythoncentral.io/pythons-range-function-explained/

File Import

As mentioned in the previous post, this post will be about how to import a csv file into the Python environment.

I started by creating an excel file called ClaimsToday.xls containing fictitious information for 6 different claims. In order to transition smoothly into the checks for validity that will be addressed in the next post, I configured the below data such that out of the six claims, only one claim has all valid data fields:



Note the following:

• Claim 123456 has the email field missing
• Claim 121212 has the l_name field missing
• Claim 123123 has f_name and email missing
• Claim 131313 is the one valid claim with all information correct
• Claim 1234 has an invalid 4 digit claim number
• Claim 111007 has a string in the phone field, where there needs to be a number

Given this data, moving forward, the program will need to flag all claims except claim 131313, whose information the program will email. The program will print a message describing the deficiencies in the other claims and will not attempt to email their information to the claimant.

Once I created the excel file, I saved it as ClaimsToday.csv.

Below is the code to import the csv file. We import and access the data using a reader. (If you were in Advance Webdesign last semester, we learned about this in that class, but using C#!) This code recognizes the first row of our data as fieldnames by default. After reading the file, I have a print statement to verify that my code worked the way I wanted it to.

Here's the result of the print statement:

Now that we have the data we need, in my next post I will be moving forward into showing you how we can check the data before sending out the email.


Sources:
https://docs.python.org/3/library/csv.html#csv.DictReader

Project Overview

Upon my graduation this May, I will be working at Progressive Insurance as a data analyst. From what I know, I will be working with insurance claims data in some capacity.

Throughout the next three posts, I will be building a Python program that does the following. This small program will simulate a large scale program that could be used by a major insurance company to be run daily and send out emails to claimants with the information for the claim they filed that day.

1. The program will read a csv file ClaimsToday with claims information. Every row will hold information for a different claim. In reality, for a big insurance company, there would be thousands of rows, but my file will contain a small sample of claims. The various columns will hold the different pieces of information for each claim, including claim number, claimant first name, claimant last name, address, phone number, email, description, damage, location of accident, date of accident, time of accident, police involvement etc.
2. Then the program will check the data for completeness and validity. Once the data is read into the program, we want to make sure that it is clean. Certain required fields must be filled, such as claim number, first and last name, and email. Other fields will have to be checked for formatting, for instance, if phone number is not a number or if claim number is not in the right format (we'll set the standard for a claim number to be six digits for the purpose of this exercise), the program should produce an error message to report the deficiencies in the data. Note that large corporations have entire departments dedicated to data cleansing, so we will be doing a very small example of this sort of work here.
3. Finally, if the data is clean, the program will format a message with the information for each claim, which will be automatically sent to the email associated with the claim. I will be sending all emails to my personal email account for the purposes of testing the program.

My next blog post will cover the first step of this process, reading the data, so stay tuned!

Fun fact, I was Flo from Progressive this past Halloween! Until next time!

Dynamic Typing (and a new datatype!)

As mentioned before, Python is a dynamically typed language. In this post we will further explore the functionality of a dynamically typed language.

In Python, there is a cool module called "types" that defines the names of the different types of variables. We can explore this module to see how Python recognizes a variable type without it being declared by the programmer.

Identifying a datatype in Python takes a very small amount of code; this hearkens back to my previous blog post that talk about the conciseness of Python's language in one-liners of code. Because Python is dynamically typed, the "type" module is able to recognize any valid datatype for what it it.



An here is the output:



Dictionaries:

Now I will briefly introduce a datatype you may have not seen before. "A Python dictionary is a mapping of unique keys to values." Dictionaries sort of act like two lists joined together or a 2D array, where there is an association between the two parts. I am introducing dictionaries, not only because they are an interesting and useful datatype to know about and one I have not encountered before in other languages, but because I will elaborate on the above example where we recognize the datatypes of each variable to show that even more complex datatypes are recognized.

Here is the example again, with an example for a dictionary as well as a list included:


























Other than the possibly unfamiliar syntax of a dictionary, you might notice something else unfamiliar here. In this example, I declared name_number as a dictionary and primes as a list. Since Python is dynamically typed, I did not need to do this, but I wanted to demonstrate that it is still possible and could be useful to do, especially as your programs get more complicated.

I could have just as easily done the following:







Sources:
http://www.secnetix.de/olli/Python/dynamic_typing.hawk
https://medium.com/@ageitgey/learn-how-to-use-static-type-checking-in-python-3-6-in-10-minutes-12c86d72677b
http://www.pythonforbeginners.com/dictionary/how-to-use-dictionaries-in-python
http://developer.rhino3d.com/guides/rhinopython/python-datatypes/#tuple
Explore this last link to read about tuples, another datatype in Python. Tuples look and act very much like lists, but there are some important differences. I included this reference as supplemental reading as I feel it is important information, but does not necessarily relate to this post.

Python One Liners

Many code structures that take multiple lines in a language like Java can be written using only one line of Python code. One liners of code save tons of space. They also show examples of Python's concise and structured language as I described in the first blog post.

We have already discussed how brief language is a special feature of Python, such as one line conditional statements. In this post, I will list some more examples of how Python uses one line of code to do things that take multiple lines in a language such as Java.

1. Swapping numbers

It's the tricky question that's bound to be on an exam in your coding 101 class. In Java, swapping numbers requires something like the following code, where a temp variable is created to store one of the variables before swapping:



In Python, however, we can simply use one line of code, rather than three to accomplish the swap. This is called 'unpacking for swapping variables':







2. Duplicating strings

In a language like Java, to duplicated a string, one would have to write code in a loop to concatenate a string:





But in Python, we can "multiply" a string like we would a number, all in just one line of code.





3. Store list values in separate variables

In Python, we can assign values in a list to variables using only one line of code.





Doing this in Java would require a loop to go through each position in the array. I will not show this code in Java, but you can imagine that it would take many more lines of code than we used in Python.

4. Combine items in 2 separate lists

Items in two lists can be joined in their respective positions using one line of Python code. The following is an example. Note that in this example, we use a two-liner for loop to 'zip' the lists together.








Doing this in another language would require looping through each list and concatenating the strings position by position, which would certainly take more than two lines of code.

5. Negative Indexing

Another unique feature to Python is negative indexing. Python recognizes negative indexes of a lists, and interprets them by counting through the list backwards. This is a cool feature that can be coded in one line, and it is not a feature in languages like Java.





And now for a (Monty) Python one-liner:
"My brain hurts!"
- Monty Python's Flying Circus

Until next week!

Sources:
https://www.codementor.io/sumit12dec/python-tricks-for-beginners-du107t193
http://sahandsaba.com/thirty-python-language-features-and-tricks-you-may-not-know.html

Binary Search

Concept 

A binary search reads through a sorted list of numbers finding midpoints in order to find a particular number's position in the list. The search compares whether the desired number is equal, less than, or greater than the midpoint. If the number is equal to the midpoint, the search process is done. If the number is less than the midpoint, in the next iteration of the search, we cut off all numbers above the midpoint before calculating a new midpoint. Similarly, if the number is greater than the midpoint, we cut off all numbers less than the midpoint for the next iteration. We search until the number is equal to the midpoint and return the position of that number in the list. Otherwise, the number is not in the list.

Code

The code below is one version of binary search in Python. There are comments in the code itself, but I will draw your attention to the structure in detail. The following is a more expanded version of the Concept above:

We use a method that takes in a list and a number as inputs.

We define the original first position as position 0 and the last position as the length of the list minus 1.

We use a while loop to search through the list as long as (considering that the 'first' and 'last' positions in the list narrow down the search) first position is less than or equal to the last.

In the loop, we designate the midpoint to be between the first and last positions in the list.

If the desired number equals the midpoint, we have found the position! We return the result.

Note that .format(...) assigns variables to the bracketed words in the quotations. (This is a different way of writing a string with variables than we have seen before.)

Then, if the item in the list does not equal the midpoint, if it is less than the item in the midpoint position in the list, we change 'last' to one less than the midpoint, thereby cutting off the upper half of the list for the purpose of finding the midpoint in the next iteration of the loop.

Similarly, if the item is greater, we cut off the lower half of the list for the next iteration.

If the loop ends before returning a statement, we conclude that the item in the list is not found.

We used multiple Python features in this program that we have learned previously:

Methods

Loops

Conditionals

Variables

Arithmetic operations

:)

Sources:

I combined the following two sources in my own code for binary search

http://interactivepython.org/runestone/static/pythonds/SortSearch/TheBinarySearch.html

https://gist.github.com/JonathanSpeek/1f4c7c283c7c3c475ee13d57381765d8

Methods

Methods in Python have both similarities and differences to Java and other languages. Now that we have explored the concepts of loops and conditionals and are aware of some Python syntax, structure, and indentation rules, looking at the nature of methods is a good next step!

Basic Structure

• First off, in Python, a class is necessary to run a method, so we must create one followed by a colon. 
• Then, after an indentation, you need to define your method using the word 'def' followed by the name of the method. After the name of the method, in parentheses, we need to type the word 'self.' By convention, this is always the first argument in a method, so whether we have arguments to pass in or not, we need 'self' to refer to the newly created object. As in other Python code structures like loops and conditionals, we end this line with a colon. 
• Following another indentation on the next line, we code what we want the method to accomplish. In the following example, I simply have a print statement, but methods can have multiple lines and statements in this section as long as they adhere to Python indentation rules.
• Finally, to call the method, we call the method within the class. The example below is a very basic picture of what a Python method looks like: 

Constructor and Instance Variables

Now that we know the basics, let's define instance variables inside methods. The  __init__ constructor method is called once for any object of a class. In this example, we add numbers by passing parameters into the constructor. We define the instance variable sum within the method and then outside of the method, we print it. 

Note that alternatively, we would get the same result as the above program without using an instance variable, but my goal with the above example was to show how instance variables can be incorporated into methods:

We can also get the same result by calling our own created method instead of the constructor. When we write a method without a constructor, Python executes a default constructor that doesn't do anything. The difference between the code below and the code right above is that we have to create an object (adding) of the Addition class. The following method follows the same formula as the Basic Structure example.

Main Method

As in other object-oriented programming languages, we can use a main method to call all of our other methods. This is the sort of structure we would want to use as our programs become more complex and developed. The example below shows some key features of the main method:

• Notice first that the final line of code calls to execute the main method. Without this, nothing happens.
• In the Dog class, we declare class variables, define a constructor method, and define another method called set_score.
• In the main method we create objects and pass in variables to run the constructor method to set up that object's instance variables.
• We can print instance variables as well as class variables for a particular object from the main method.
• We can call the method set_score in the Dog class from the main() method.

Lessons

There are different ways to do things to do things in Python. For example, instead of simply writing "main()" at the end of my program, I could have written a longer statement: 

if __name__=="__main__": main()

This is what my source was directing me to do. At first I did not understand it, so I looked up the purpose of this lengthy and confusing statement. In Python, __name__ is set to __main__ whenever a file is being executed, so when you run your program, if __name__=="__main__" returns true, which executes main(). But I found that if I eliminate the conditional my program still works. I am sure there are cases when the conditional is necessary, but in my case it was not. The lesson learned was that not everything is as complicated as sources might tell you, so it's worth monkeying around on your own to find what might be a simpler and more understandable version of your code.

Also, when I was putting this together, I was getting quite a few errors for various little things here and there. I think most of them were related to a struggle to remember how object oriented programming works, so if you've been away from this type of programming, I hope this post was a nice refresher!

Sources:

https://stackoverflow.com/questions/3786881/what-is-a-method-in-python
https://www.digitalocean.com/community/tutorials/understanding-class-and-instance-variables-in-python-3
https://stackoverflow.com/questions/22492162/understanding-the-main-method-of-python

Repeated Action Syntax

There are two types of loops in Python, the for loop and the while loop.

Important to note: In loops, as in conditional statements, indentation matters.

For Loop:

The syntax for for-loops is the following:

The following example shows how we can print each letter of a word, using a loop. Since Python is dynamically typed, the program recognizes that word is a string variable, so the loop prints every letter of word. This is an example of strings as an iterable:

                

We can do something similar with numbers, looping through an array:

We can also nest conditionals inside of loops. Python uses "continue" and "break" statements to control the flow of the loop; the word "continue" causes the loop to skip the current block and return to the top for or while statement, whereas "break" stops the loop. These key words are not mandatory for loops and they can be used alone or together; they can be very useful and provide a quick solution in certain circumstances and a feature that other languages lack. In the example below, we skip even numbers and print the odd numbers in a range from 1-10:

The next example adds two lines to the code above to demonstrate how we can end a loop. Here we end the loop before printing the number 9:

While Loop:

While loops in Python follow a similar syntax to for loops:

In the following program, we add two at the end of each loop, resulting in even numbers 0-8:

        

We can use "continue" and/or "break" in while loops as well:

         

Nested Loops:

In Python, we can nest loops. The following program is an algorithm that prints prime numbers from one to one hundred. The outer loop runs through integers (i) two to ninety-nine. The inner loop loops through integers (j) that could be possible divisors of (i). If (j) does not divide (i) and (j) is greater than (i/j), (i) is prime. Until we reach a (j) that divides (i), we loop until (j) is greater than (i/j), so (i) is prime.

                                

                      

We can nest for loops in a similar way. The following example is a simple one that I made up to multiply numbers zero through nine by each number in the list containing one, two, and three:

             

         

It is also possible to put a for loop in a while loop and vice versa. Often times, we use nested loops to cycle through items in a list. The following method demonstrates this possibility in Python. Note how the conditional with the boolean here acts like the 'break' statement described previously to end the iterations of the loop:

In coding, there is usually more use for a while loop inside of a for loop than a for loop inside of a while loop, so I will not provide an example of the latter, but it is possible.

Else Statements in Loops:

Unlike languages like C, in Python, we can execute an "else" statement upon the completion of a loop. In the second loop below, the the loop is terminated before the end of it's iterations, so the else statement does not get executed.

   

Else statements in loops were a new concept to me, and they seem to be useful checks for when a loop has completed its rounds.

Also, in Python loops can be infinite. The following program will run forever unless you stop it, so be careful!

Response to Michelle's first comment:

Great question! In a 'for’ loop, to increment by a number other than one, you add parameters to range. The following code shows an example of this. The 'step' is whatever increment you want to add through every iteration of the loop:

             

Note if you were to use a while loop, the syntax to increment a variable would be: "i += number_you_want_to_add," and examples of this are included in my post.

Sources:

https://swcarpentry.github.io/python-novice-inflammation/02-loop/
https://www.learnpython.org/en/Loops
https://www.tutorialspoint.com/python/python_nested_loops.htm
https://stackoverflow.com/questions/21930663/while-loop-inside-of-for-loop