More efficient way to read large scales of Mails in Outlook using Python

Data Science Python
,
1

Hi Experts.

Below is the python code which I’m using for reading mails from a mailbox.

I’m reading mails from Inbox and Subfolder. In this I had around 25 thousands of mails, for this it is taking 16 minutes.

My question is: is there any efficient way in python code to get the results within short time?
If is there any answers that will very helpful and appreciated.

import win32com.client as client
import pandas as pd
import time

start_time = time.time()
#messages.Sort("[ReceivedTime]", True)

outlook = client.Dispatch("Outlook.Application").GetNamespace("MAPI")

# XXXXXXXXXXXXXX == mentioning mail ID here
#Accessing mail box
root_folder = outlook.Folders.Item('XXXXXX')

#inbox messges
print("reading Inbox...")
Inbox = root_folder.Folders['Inbox']
inbox_messages = Inbox.items
inbox_messages.Sort("[ReceivedTime]", True)

#Accessing particular folder 'Redeployment' in Inbox
print("reading Inbox Sub folder...")
#accessing subfolder inside the Inbox
Inbox_subfolder = Inbox.Folders['xxxxxxxxxx']
subfolder_messages = Inbox_subfolder.items
subfolder_messages.Sort("[ReceivedTime]", True)

#list to store inbox mails in dataframe
inbox_list = []
#list to store subfolder mails in dataframe
subfolder_list = []
#iterating inbox messages
print("Retrieving mails from Inbox")
for message in inbox_messages:
    print(f"message count: {len(message)}")
    if message.Class == 43:
        received_date = message.ReceivedTime.strftime('%d-%m-%Y %I:%M:%S')
        inbox_list.append((message.Subject,message.Sender,message.To,received_date,message.body,message,Inbox))
#Inbox messages to dataframe
inbox_df = pd.DataFrame(inbox_list)
inbox_df.columns =['Subject','Sender', 'Receipients','Received Date','Mail Body','Message','Folder']
mid_time = time.time()
print("Reading Inbox mails time taken is {}\n\n".format(mid_time-start_time))
#iterating inbox messages
print("\nRetrieving mails from subfolder\n")
for message in subfolder_messages:
    if message.Class == 43:
        received_date = message.ReceivedTime.strftime('%d-%m-%Y %I:%M:%S')
        subfolder_list.append((message.Subject, message.Sender,message.To,received_date,message.body,message,Inbox_subfolder))
#subfolder messages to dataframe
subfolder_df = pd.DataFrame(subfolder_list)
subfolder_df.columns =['Subject','Sender', 'Receipients','Received Date','Mail Body','Message','Folder']
# using a .concat() method
frames = [inbox_df, subfolder_df]
final_mail_df = pd.concat(frames)

end_time = time.time()
print("\nReading subfolder mails time taken is {}\n\n".format(end_time-start_time))

#print(final_mail_df)
final_mail_df.to_excel("mail_dataframe.xlsx",index=False)
print("\nCompleted")
2

You can divide reading process to different threads (separate thread for each subfolder for example), but if bottleneck in mail server this can be not so helpful.

3

Thank you for your reply. I have got only one mail box and two folders.

  1. Inbox with 100 emails
  2. A subfolder in inbox with 25000 emails.

i need to extract the sender, recipients, mail body, subject, and the mail received time to a pandas dataframe . Currently i am doing this via a loop and it is taking too much time. Is there a efficient way to extract the information i need into a pandas Dataframe or excel in a faster way.

4

Hi, I can’t test your case, but try something like this:

import win32com.client as client
import pandas as pd
import time

import concurrent.futures

def handle_message(message, inbox_list, Inbox):
    if message.Class == 43:
        received_date = message.ReceivedTime.strftime('%d-%m-%Y %I:%M:%S')
        inbox_list.append((message.Subject,message.Sender,message.To,received_date,message.body,message,Inbox))
    return f"message count: {len(message)}"


start_time = time.time()
#messages.Sort("[ReceivedTime]", True)

outlook = client.Dispatch("Outlook.Application").GetNamespace("MAPI")

# XXXXXXXXXXXXXX == mentioning mail ID here
#Accessing mail box
root_folder = outlook.Folders.Item('XXXXXX')

#inbox messges
print("reading Inbox...")
Inbox = root_folder.Folders['Inbox']
inbox_messages = Inbox.items
inbox_messages.Sort("[ReceivedTime]", True)

#Accessing particular folder 'Redeployment' in Inbox
print("reading Inbox Sub folder...")
#accessing subfolder inside the Inbox
Inbox_subfolder = Inbox.Folders['xxxxxxxxxx']
subfolder_messages = Inbox_subfolder.items
subfolder_messages.Sort("[ReceivedTime]", True)

#list to store inbox mails in dataframe
inbox_list = []
#list to store subfolder mails in dataframe
subfolder_list = []
#iterating inbox messages
print("Retrieving mails from Inbox")

with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
    message_futures = [executor.submit(handle_message, message, inbox_list, Inbox) for message in inbox_messages]
    for future in concurrent.futures.as_completed(message_futures):
        print(future.result())

# for message in inbox_messages:
    # print(f"message count: {len(message)}")
    # if message.Class == 43:
        # received_date = message.ReceivedTime.strftime('%d-%m-%Y %I:%M:%S')
        # inbox_list.append((message.Subject,message.Sender,message.To,received_date,message.body,message,Inbox))
#Inbox messages to dataframe
inbox_df = pd.DataFrame(inbox_list)
inbox_df.columns =['Subject','Sender', 'Receipients','Received Date','Mail Body','Message','Folder']
mid_time = time.time()
print("Reading Inbox mails time taken is {}\n\n".format(mid_time-start_time))
#iterating inbox messages
print("\nRetrieving mails from subfolder\n")

with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
    message_futures = [executor.submit(handle_message, message, subfolder_list, Inbox_subfolder) for message in subfolder_messages]
    for future in concurrent.futures.as_completed(message_futures):
        print(future.result())


# for message in subfolder_messages:
    # if message.Class == 43:
        # received_date = message.ReceivedTime.strftime('%d-%m-%Y %I:%M:%S')
        # subfolder_list.append((message.Subject, message.Sender,message.To,received_date,message.body,message,Inbox_subfolder))
#subfolder messages to dataframe
subfolder_df = pd.DataFrame(subfolder_list)
subfolder_df.columns =['Subject','Sender', 'Receipients','Received Date','Mail Body','Message','Folder']
# using a .concat() method
frames = [inbox_df, subfolder_df]
final_mail_df = pd.concat(frames)

end_time = time.time()
print("\nReading subfolder mails time taken is {}\n\n".format(end_time-start_time))

#print(final_mail_df)
final_mail_df.to_excel("mail_dataframe.xlsx",index=False)
print("\nCompleted")
1 Like
5

Thank you so much for the reply. Let me test it and come back.

6

Here are a few suggestions for how you can optimize the process of extracting the information you need from your emails:

  1. Instead of using a loop to process each email individually, you could try using a batch processing approach. This could involve reading and processing multiple emails at a time, which could be more efficient than processing them one at a time.
  2. You could also try using a library or API specifically designed for reading and processing emails. For example, the imaplib library in Python provides an interface for accessing email servers and retrieving email messages, and the email library provides functions for parsing and extracting information from email messages.
  3. Another option would be to use a tool or service that is specifically designed for extracting data from emails and storing it in a structured format, such as a spreadsheet or database. These tools often provide pre-built connectors or integrations for popular email platforms, which can make the process of extracting and storing data more efficient.
1 Like