samedi 14 novembre 2020

Pythonic way to avoid using multiple IF statements when iterating and appending to a Pandas data frame

I'm retrieving data from a Jira database, then saving the data to a Pandas data frame. Here's my code:

from jira import JIRA
import pandas as pd

cert_path = 'C:\\cert.crt'

start_date = '2020-10-01'
end_date = '2020-10-31'

# three different instances (each with their own schema)
a_session = JIRA(server='https://jira.myinstance-A.com', options={'verify': cert_path}, kerberos=True)

b_session = JIRA(server='https://jira.myinstance-B.com', options={'verify': cert_path}, kerberos=True)

c_session = JIRA(server='https://jira.myinstance-C.com', options={'verify': cert_path}, kerberos=True)


# define queries
query_1 = 'project = \"Test Project 1\" and issuetype = Incident and resolution = Resolved and updated >= {} and updated <= {}'.format(start_date, end_date)

query_2 = 'project = \"Test Project 2\" and issuetype = Incident and resolution = Resolved and updated >= {} and updated <= {}'.format(start_date, end_date)

query_3 = 'project = \"Test Project 3\" and issuetype = Defect and resolution = Resolved and releasedate >= {} and releasedate <= {}'.format(start_date, end_date)

query_4 = 'project = \"Test Project 4\" and issuetype = Enhancement and resolution = Done and completed >= {} and completed <= {}'.format(start_date, end_date)


# fetch all issues from a given session for a given query
block_size = 100
block_num = 0


def get_all_issues(session, query):

    block_size = 50
    block_num = 0
    
    start = 0
    all_issues = []
    while True:
        issues = session.search_issues(query, start, block_size)
        if len(issues) == 0:
            # No more issues
            break
        start += len(issues)
        for issue in issues:
            all_issues.append(issue)

    issues = pd.DataFrame(issues)

    for issue in all_issues:
        d = {
            'key' : issue.key,
            'type' : issue.fields.type,
            'creator' : issue.fields.creator,
            'resolution' : issue.fields.resolution
             }

        issues = issues.append(d, ignore_index=True)

    return issues


# list of queries, and the corresponding backend
queries = [
    (a_session, query_1),
    (a_session, query_2),
    (b_session, query_3),
    (c_session, query_4),
]


# loop over each pair of session and query, calling the get_all_issues function, and save the dataframe we get each time
dataframes = []

for session, query in queries:
    dataframe = get_all_issues(session, query)
    dataframes.append(dataframe)

# concatenate all data frames

all = pd.concat(dataframes)

This code works just fine (because the 4 field names in the d dict are common to a_session, b_session, and c_session).

The problem arises when I try to introduce a custom field that might be present in, say, a_session but not in b_session or c_session.

For example:

for issue in all_issues:
    d = {
        'key' : issue.key,
        'type' : issue.fields.type,
        'creator' : issue.fields.creator,
        'resolution' : issue.fields.resolution,
        'system_change' : issue.fields.custom_field_123,  # only applicable to a_session and b_session
        'system_resources' : issue.fields.custom_field_456,  # only applicable to c_session
        'system_backup' : issue.fields.custom_field_789   # only applicable to b_session and c_session
         }

custom_field_123 exists in a_session and b_session, but not in c_session.

custom_field_456 exists only in c_session.

And, custom_field_789 exists in b_session and c_session.

Running the code with this expanded dictionary results in the following error: AttributeError: type object 'PropertyHolder' has no attribute 'custom_field_123'.

Other than using IF statements, is there a Pythonic way to allocate the only the valid field names in the d dictionary to the relevant sessions? For example, as a_session/query_2 is running, only consider custom_field_123 (because that's the only one that's valid) and DISREGARD custom_field_456 and custom_field_789 (because passing these would result in the error message above).

Thanks in advance for any help you can give this Python novice!

Aucun commentaire:

Enregistrer un commentaire