diff --git a/lab-dw-pandas.ipynb b/lab-dw-pandas.ipynb new file mode 100644 index 0000000..d3d9c0b --- /dev/null +++ b/lab-dw-pandas.ipynb @@ -0,0 +1,754 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "25d7736c-ba17-4aff-b6bb-66eba20fbf4e", + "metadata": {}, + "source": [ + "# Lab | Pandas" + ] + }, + { + "cell_type": "markdown", + "id": "d1973e9e-8be6-4039-b70e-d73ee0d94c99", + "metadata": {}, + "source": [ + "In this lab, we will be working with the customer data from an insurance company, which can be found in the CSV file located at the following link: https://raw.githubusercontent.com/data-bootcamp-v4/data/main/file1.csv\n", + "\n", + "The data includes information such as customer ID, state, gender, education, income, and other variables that can be used to perform various analyses.\n", + "\n", + "Throughout the lab, we will be using the pandas library in Python to manipulate and analyze the data. Pandas is a powerful library that provides various data manipulation and analysis tools, including the ability to load and manipulate data from a variety of sources, including CSV files." + ] + }, + { + "cell_type": "markdown", + "id": "8045146f-f4f7-44d9-8cd9-130d6400c73a", + "metadata": {}, + "source": [ + "### Data Description\n", + "\n", + "- Customer - Customer ID\n", + "\n", + "- ST - State where customers live\n", + "\n", + "- Gender - Gender of the customer\n", + "\n", + "- Education - Background education of customers \n", + "\n", + "- Customer Lifetime Value - Customer lifetime value(CLV) is the total revenue the client will derive from their entire relationship with a customer. In other words, is the predicted or calculated value of a customer over their entire duration as a policyholder with the insurance company. It is an estimation of the net profit that the insurance company expects to generate from a customer throughout their relationship with the company. Customer Lifetime Value takes into account factors such as the duration of the customer's policy, premium payments, claim history, renewal likelihood, and potential additional services or products the customer may purchase. It helps insurers assess the long-term profitability and value associated with retaining a particular customer.\n", + "\n", + "- Income - Customers income\n", + "\n", + "- Monthly Premium Auto - Amount of money the customer pays on a monthly basis as a premium for their auto insurance coverage. It represents the recurring cost that the insured person must pay to maintain their insurance policy and receive coverage for potential damages, accidents, or other covered events related to their vehicle.\n", + "\n", + "- Number of Open Complaints - Number of complaints the customer opened\n", + "\n", + "- Policy Type - There are three type of policies in car insurance (Corporate Auto, Personal Auto, and Special Auto)\n", + "\n", + "- Vehicle Class - Type of vehicle classes that customers have Two-Door Car, Four-Door Car SUV, Luxury SUV, Sports Car, and Luxury Car\n", + "\n", + "- Total Claim Amount - the sum of all claims made by the customer. It represents the total monetary value of all approved claims for incidents such as accidents, theft, vandalism, or other covered events.\n" + ] + }, + { + "cell_type": "markdown", + "id": "3a72419b-20fc-4905-817a-8c83abc59de6", + "metadata": {}, + "source": [ + "External Resources: https://towardsdatascience.com/filtering-data-frames-in-pandas-b570b1f834b9" + ] + }, + { + "cell_type": "markdown", + "id": "8f8ece17-e919-4e23-96c0-c7c59778436a", + "metadata": {}, + "source": [ + "## Challenge 1: Understanding the data\n", + "\n", + "In this challenge, you will use pandas to explore a given dataset. Your task is to gain a deep understanding of the data by analyzing its characteristics, dimensions, and statistical properties." + ] + }, + { + "cell_type": "markdown", + "id": "91437bd5-59a6-49c0-8150-ef0e6e6eb253", + "metadata": {}, + "source": [ + "- Identify the dimensions of the dataset by determining the number of rows and columns it contains.\n", + "- Determine the data types of each column and evaluate whether they are appropriate for the nature of the variable. You should also provide suggestions for fixing any incorrect data types.\n", + "- Identify the number of unique values for each column and determine which columns appear to be categorical. You should also describe the unique values of each categorical column and the range of values for numerical columns, and give your insights.\n", + "- Compute summary statistics such as mean, median, mode, standard deviation, and quartiles to understand the central tendency and distribution of the data for numerical columns. You should also provide your conclusions based on these summary statistics.\n", + "- Compute summary statistics for categorical columns and providing your conclusions based on these statistics." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "dd4e8cd8-a6f6-486c-a5c4-1745b0c035f4", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "0c468f70", + "metadata": {}, + "outputs": [], + "source": [ + "customer_data_insurance_df = pd.read_csv(\"Customer_data.csv\")" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "bbee74f1", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The dataset has 4008 rows and 11 columns.\n" + ] + } + ], + "source": [ + "rows, columns = customer_data_insurance_df.shape\n", + "print(f\"The dataset has {rows} rows and {columns} columns.\")" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "65b9a8b3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "RangeIndex: 4008 entries, 0 to 4007\n", + "Data columns (total 11 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 Customer 1071 non-null object \n", + " 1 ST 1071 non-null object \n", + " 2 GENDER 954 non-null object \n", + " 3 Education 1071 non-null object \n", + " 4 Customer Lifetime Value 1068 non-null object \n", + " 5 Income 1071 non-null float64\n", + " 6 Monthly Premium Auto 1071 non-null float64\n", + " 7 Number of Open Complaints 1071 non-null object \n", + " 8 Policy Type 1071 non-null object \n", + " 9 Vehicle Class 1071 non-null object \n", + " 10 Total Claim Amount 1071 non-null float64\n", + "dtypes: float64(3), object(8)\n", + "memory usage: 344.6+ KB\n" + ] + } + ], + "source": [ + "customer_data_insurance_df.info()" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "0c5c085d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Customer object\n", + "ST object\n", + "GENDER object\n", + "Education object\n", + "Customer Lifetime Value object\n", + "Income float64\n", + "Monthly Premium Auto float64\n", + "Number of Open Complaints object\n", + "Policy Type object\n", + "Vehicle Class object\n", + "Total Claim Amount float64\n", + "dtype: object\n", + "Consider converting Customer Lifetime Value to float for monetary values.\n", + "Consider converting Number of Open Complaints to int for count values.\n" + ] + } + ], + "source": [ + "print(customer_data_insurance_df.dtypes)\n", + "\n", + "for column in customer_data_insurance_df.columns:\n", + " if column == 'Customer Lifetime Value' and customer_data_insurance_df[column].dtype != 'float64':\n", + " print(f\"Consider converting {column} to float for monetary values.\")\n", + " elif column == 'Number of Open Complaints' and customer_data_insurance_df[column].dtype != 'int64':\n", + " print(f\"Consider converting {column} to int for count values.\")" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "070528f0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Customer has 1071 unique values.\n", + "ST has 8 unique values.\n", + "ST appears to be categorical.\n", + "Unique values: ['Washington' 'Arizona' 'Nevada' 'California' 'Oregon' 'Cali' 'AZ' 'WA'\n", + " nan]\n", + "GENDER has 5 unique values.\n", + "GENDER appears to be categorical.\n", + "Unique values: [nan 'F' 'M' 'Femal' 'Male' 'female']\n", + "Education has 6 unique values.\n", + "Education appears to be categorical.\n", + "Unique values: ['Master' 'Bachelor' 'High School or Below' 'College' 'Bachelors' 'Doctor'\n", + " nan]\n", + "Customer Lifetime Value has 1027 unique values.\n", + "Income has 774 unique values.\n", + "Range for Income: 0.0 to 99960.0\n", + "Monthly Premium Auto has 132 unique values.\n", + "Range for Monthly Premium Auto: 61.0 to 35354.0\n", + "Number of Open Complaints has 6 unique values.\n", + "Number of Open Complaints appears to be categorical.\n", + "Unique values: ['1/0/00' '1/2/00' '1/1/00' '1/3/00' '1/5/00' '1/4/00' nan]\n", + "Policy Type has 3 unique values.\n", + "Policy Type appears to be categorical.\n", + "Unique values: ['Personal Auto' 'Corporate Auto' 'Special Auto' nan]\n", + "Vehicle Class has 6 unique values.\n", + "Vehicle Class appears to be categorical.\n", + "Unique values: ['Four-Door Car' 'Two-Door Car' 'SUV' 'Luxury SUV' 'Sports Car'\n", + " 'Luxury Car' nan]\n", + "Total Claim Amount has 761 unique values.\n", + "Range for Total Claim Amount: 0.382107 to 2893.239678\n" + ] + } + ], + "source": [ + "for column in customer_data_insurance_df.columns:\n", + " unique_values = customer_data_insurance_df[column].nunique()\n", + " print(f\"{column} has {unique_values} unique values.\")\n", + " if unique_values < len(customer_data_insurance_df) * 0.02:\n", + " print(f\"{column} appears to be categorical.\")\n", + " print(\"Unique values: \", customer_data_insurance_df[column].unique())\n", + " elif customer_data_insurance_df[column].dtype in ['int64', 'float64']:\n", + " print(f\"Range for {column}: {customer_data_insurance_df[column].min()} to {customer_data_insurance_df[column].max()}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "7bb993ad", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Summary statistics for Income:\n", + "Mean: 39295.70121381886, Median: 36234.0\n", + "Mode: 0.0\n", + "Standard Deviation: 30469.42706040185\n", + "Quartiles: \n", + "0.25 14072.0\n", + "0.50 36234.0\n", + "0.75 64631.0\n", + "Name: Income, dtype: float64\n", + "\n", + "Summary statistics for Monthly Premium Auto:\n", + "Mean: 193.234360410831, Median: 83.0\n", + "Mode: 65.0\n", + "Standard Deviation: 1601.190368577621\n", + "Quartiles: \n", + "0.25 68.0\n", + "0.50 83.0\n", + "0.75 109.5\n", + "Name: Monthly Premium Auto, dtype: float64\n", + "\n", + "Summary statistics for Total Claim Amount:\n", + "Mean: 404.98690940896364, Median: 354.729129\n", + "Mode: 321.6\n", + "Standard Deviation: 293.0272599264041\n", + "Quartiles: \n", + "0.25 202.157702\n", + "0.50 354.729129\n", + "0.75 532.800000\n", + "Name: Total Claim Amount, dtype: float64\n", + "\n" + ] + } + ], + "source": [ + "for column in customer_data_insurance_df.select_dtypes(include=['int64', 'float64']).columns:\n", + " print(f\"Summary statistics for {column}:\")\n", + " print(f\"Mean: {customer_data_insurance_df[column].mean()}, Median: {customer_data_insurance_df[column].median()}\")\n", + " print(f\"Mode: {customer_data_insurance_df[column].mode()[0]}\")\n", + " print(f\"Standard Deviation: {customer_data_insurance_df[column].std()}\")\n", + " print(f\"Quartiles: \\n{customer_data_insurance_df[column].quantile([0.25, 0.5, 0.75])}\")\n", + " print()" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "07ee247e", + "metadata": {}, + "outputs": [], + "source": [ + "#Key Insights and Actions:\n", + "#Data Quality: The mode of 0 in Income likely indicates placeholders or missing data, potentially skewing results.\n", + "#Distributions: Large standard deviations and mean-median gaps, especially in Monthly Premium Auto, suggest skewed data with outliers.\n", + "#Recommended Actions: Investigate whether zero modes are placeholders or errors. Use visual tools like box plots to assess outliers and distribution skews.\n", + "#Consider data cleaning strategies, such as filtering or imputing missing values." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "a987eaec", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Value counts for categorical column Customer:\n", + "Customer\n", + "RB50392 1\n", + "QZ44356 1\n", + "AI49188 1\n", + "WW63253 1\n", + "GA49547 1\n", + " ..\n", + "TM65736 1\n", + "VJ51327 1\n", + "GS98873 1\n", + "CW49887 1\n", + "MY31220 1\n", + "Name: count, Length: 1071, dtype: int64\n", + "\n", + "Value counts for categorical column ST:\n", + "ST\n", + "Oregon 320\n", + "California 211\n", + "Arizona 186\n", + "Cali 120\n", + "Nevada 98\n", + "Washington 81\n", + "WA 30\n", + "AZ 25\n", + "Name: count, dtype: int64\n", + "\n", + "Value counts for categorical column GENDER:\n", + "GENDER\n", + "F 457\n", + "M 413\n", + "Male 39\n", + "female 28\n", + "Femal 17\n", + "Name: count, dtype: int64\n", + "\n", + "Value counts for categorical column Education:\n", + "Education\n", + "Bachelor 324\n", + "College 313\n", + "High School or Below 296\n", + "Master 94\n", + "Doctor 37\n", + "Bachelors 7\n", + "Name: count, dtype: int64\n", + "\n", + "Value counts for categorical column Customer Lifetime Value:\n", + "Customer Lifetime Value\n", + "445811.34% 4\n", + "251459.20% 4\n", + "578018.22% 3\n", + "2412750.40% 3\n", + "684615.03% 3\n", + " ..\n", + "245357.08% 1\n", + "507566.27% 1\n", + "321497.94% 1\n", + "1227534.31% 1\n", + "2344490.05% 1\n", + "Name: count, Length: 1027, dtype: int64\n", + "\n", + "Value counts for categorical column Number of Open Complaints:\n", + "Number of Open Complaints\n", + "1/0/00 830\n", + "1/1/00 138\n", + "1/2/00 50\n", + "1/3/00 34\n", + "1/4/00 13\n", + "1/5/00 6\n", + "Name: count, dtype: int64\n", + "\n", + "Value counts for categorical column Policy Type:\n", + "Policy Type\n", + "Personal Auto 780\n", + "Corporate Auto 234\n", + "Special Auto 57\n", + "Name: count, dtype: int64\n", + "\n", + "Value counts for categorical column Vehicle Class:\n", + "Vehicle Class\n", + "Four-Door Car 576\n", + "Two-Door Car 205\n", + "SUV 199\n", + "Sports Car 57\n", + "Luxury SUV 20\n", + "Luxury Car 14\n", + "Name: count, dtype: int64\n", + "\n" + ] + } + ], + "source": [ + "for column in customer_data_insurance_df.select_dtypes(include=['object']).columns:\n", + " print(f\"Value counts for categorical column {column}:\")\n", + " print(customer_data_insurance_df[column].value_counts())\n", + " print()" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "67244564", + "metadata": {}, + "outputs": [], + "source": [ + "#Conclusions from Categorical Data:\n", + "#Customer: Each customer ID ('RB50392', 'QZ44356', etc.) occurs exactly once.\n", + "#Conclusion: Customer IDs are unique identifiers, useful for distinguishing individual records but not suitable for grouping or statistical analysis.\n", + "#ST (State): Some states are represented by both full names and abbreviations, e.g., 'California' and 'Cali', 'Washington' and 'WA', 'Arizona' and 'AZ'.\n", + "#Conclusion: Consider standardizing state names to enhance consistency and accuracy in analysis or reporting (e.g., converting all to full names or all to abbreviations).\n", + "#GENDER: Variations in gender labeling exist, such as 'F', 'female', 'Femal', 'M', and 'Male'.\n", + "#Conclusion: Normalize gender entries for accurate subgroup comparison and clarity, i.e., consolidate labels into standard categories like 'Male' and 'Female'.\n", + "#Education: The dataset includes categories like 'Bachelor's' and 'Bachelors'.\n", + "#Conclusion: Harmonize similar educational attainment labels to facilitate accurate grouping and analysis, ensuring there is no redundancy with slightly different spelling.\n", + "#Customer Lifetime Value: This should be numeric, but the presence of percentages might indicate a misformatting or incorrect labeling.\n", + "#Conclusion: Correct data type and formatting matters here. Remove % signs or convert CLV appropriately to numerical values for analysis, as values mean meaningful numeric data viewing, not percentages.\n", + "#Number of Open Complaints: Observation: Coded in the format '1/x/00', this might represent encoded information related to the number of complaints, not ideal for a categorical representation explicitly.\n", + "#Conclusion: Understand the structure, making it easier to interpret, typically favoring clear separation into columns rather than complex encoding.\n", + "#Policy Type: Categories such as 'Personal Auto', 'Corporate Auto', and 'Special Auto' are distinctly separated.\n", + "#Conclusion: Clear distinction in policy type allows for straightforward analysis of differences in policy impacts or preferences.\n", + "#Vehicle Class: Categories indicate car types with 'Four-Door Car' showing high frequency.\n", + "#Conclusion: These labels appear useful for understanding vehicle distribution and potential product preferences or needs.\n", + "#Recommendations:\n", + "#Data Cleaning and Normalization: Address inconsistencies in categorical labels to prevent fragmentation in analyses. This is essential for comparability and clarity across entries.\n", + "#Ensure Correct Typing: Misformatted numerical information labeled as categorical could mask clear insights or affect numerical computations—correct as needed.\n", + "#Review and Recode as Needed: Relevant for Number of Open Complaints, the encoding may require clearer labels/methodology for user understanding since it presents key operational metrics" + ] + }, + { + "cell_type": "markdown", + "id": "4a703890-63db-4944-b7ab-95a4f8185120", + "metadata": {}, + "source": [ + "## Challenge 2: analyzing the data" + ] + }, + { + "cell_type": "markdown", + "id": "0776a403-c56a-452f-ac33-5fd4fdb06fc7", + "metadata": {}, + "source": [ + "### Exercise 1" + ] + }, + { + "cell_type": "markdown", + "id": "eedbc484-da4d-4f9c-9343-e1d44311a87e", + "metadata": {}, + "source": [ + "The marketing team wants to know the top 5 less common customer locations. Create a pandas Series object that contains the customer locations and their frequencies, and then retrieve the top 5 less common locations in ascending order." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "2dca5073-4520-4f42-9390-4b92733284ed", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "ST\n", + "AZ 25\n", + "WA 30\n", + "Washington 81\n", + "Nevada 98\n", + "Cali 120\n", + "Name: count, dtype: int64\n" + ] + } + ], + "source": [ + "location_counts = customer_data_insurance_df[\"ST\"].value_counts()\n", + "\n", + "least_common_locations = location_counts.nsmallest(5).sort_values(ascending=True)\n", + "\n", + "print(least_common_locations)" + ] + }, + { + "cell_type": "markdown", + "id": "0ce80f43-4afa-43c7-a78a-c917444da4e0", + "metadata": {}, + "source": [ + "### Exercise 2\n", + "\n", + "The sales team wants to know the total number of policies sold for each type of policy. Create a pandas Series object that contains the policy types and their total number of policies sold, and then retrieve the policy type with the highest number of policies sold." + ] + }, + { + "cell_type": "markdown", + "id": "a9f13997-1555-4f98-aca6-970fda1d2c3f", + "metadata": {}, + "source": [ + "*Hint:*\n", + "- *Using value_counts() method simplifies this analysis.*\n", + "- *Futhermore, there is a method that returns the index of the maximum value in a column or row.*\n" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "bcfad6c1-9af2-4b0b-9aa9-0dc5c17473c0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Total number of policies sold for each type:\n", + "Policy Type\n", + "Personal Auto 780\n", + "Corporate Auto 234\n", + "Special Auto 57\n", + "Name: count, dtype: int64\n", + "\n", + "Policy type with the highest number of policies sold: Personal Auto\n" + ] + } + ], + "source": [ + "policy_counts = customer_data_insurance_df['Policy Type'].value_counts()\n", + "\n", + "print(\"Total number of policies sold for each type:\")\n", + "print(policy_counts)\n", + "\n", + "top_policy_type = policy_counts.idxmax()\n", + "\n", + "print(\"\\nPolicy type with the highest number of policies sold:\", top_policy_type)" + ] + }, + { + "cell_type": "markdown", + "id": "0b863fd3-bf91-4d5d-86eb-be29ed9f5b70", + "metadata": {}, + "source": [ + "### Exercise 3\n", + "\n", + "The sales team wants to know if customers with Personal Auto have a lower income than those with Corporate Auto. How does the average income compare between the two policy types?" + ] + }, + { + "cell_type": "markdown", + "id": "b1386d75-2810-4aa1-93e0-9485aa12d552", + "metadata": {}, + "source": [ + "- Use *loc* to create two dataframes: one containing only Personal Auto policies and one containing only Corporate Auto policies.\n", + "- Calculate the average income for each policy.\n", + "- Print the results." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "0c0563cf-6f8b-463d-a321-651a972f82e5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Average income for Personal Auto: 38180.69871794872\n", + "Average income for Corporate Auto: 41390.31196581197\n" + ] + } + ], + "source": [ + "personal_auto_df = customer_data_insurance_df.loc[customer_data_insurance_df['Policy Type'] == 'Personal Auto']\n", + "corporate_auto_df = customer_data_insurance_df.loc[customer_data_insurance_df['Policy Type'] == 'Corporate Auto']\n", + "\n", + "average_income_personal = personal_auto_df['Income'].mean()\n", + "average_income_corporate = corporate_auto_df['Income'].mean()\n", + "\n", + "print(f\"Average income for Personal Auto: {average_income_personal}\")\n", + "print(f\"Average income for Corporate Auto: {average_income_corporate}\")" + ] + }, + { + "cell_type": "markdown", + "id": "80b16c27-f4a5-4727-a229-1f88671cf4e2", + "metadata": {}, + "source": [ + "### Bonus: Exercise 4\n" + ] + }, + { + "cell_type": "markdown", + "id": "ac584986-299b-475f-ac2e-928c16c3f512", + "metadata": {}, + "source": [ + "Your goal is to identify customers with a high policy claim amount.\n", + "\n", + "Instructions:\n", + "\n", + "- Review again the statistics for total claim amount to gain an understanding of the data.\n", + "- To identify potential areas for improving customer retention and profitability, we want to focus on customers with a high policy claim amount. Consider customers with a high policy claim amount to be those in the top 25% of the total claim amount. Create a pandas DataFrame object that contains information about customers with a policy claim amount greater than the 75th percentile.\n", + "- Use DataFrame methods to calculate summary statistics about the high policy claim amount data. " + ] + }, + { + "cell_type": "markdown", + "id": "4e3af5f1-6023-4b05-9c01-d05392daa650", + "metadata": {}, + "source": [ + "*Note: When analyzing data, we often want to focus on certain groups of values to gain insights. Percentiles are a useful tool to help us define these groups. A percentile is a measure that tells us what percentage of values in a dataset are below a certain value. For example, the 75th percentile represents the value below which 75% of the data falls. Similarly, the 25th percentile represents the value below which 25% of the data falls. When we talk about the top 25%, we are referring to the values that fall above the 75th percentile, which represent the top quarter of the data. On the other hand, when we talk about the bottom 25%, we are referring to the values that fall below the 25th percentile, which represent the bottom quarter of the data. By focusing on these groups, we can identify patterns and trends that may be useful for making decisions and taking action.*\n", + "\n", + "*Hint: look for a method that gives you the percentile or quantile 0.75 and 0.25 for a Pandas Series.*" + ] + }, + { + "cell_type": "markdown", + "id": "2d234634-50bd-41e0-88f7-d5ba684455d1", + "metadata": {}, + "source": [ + "*Hint 2: check `Boolean selection according to the values of a single column` in https://towardsdatascience.com/filtering-data-frames-in-pandas-b570b1f834b9*" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "b731bca6-a760-4860-a27b-a33efa712ce0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "75th Percentile: 532.8\n", + "\n", + "Customers with high claim amounts:\n", + " Customer ST GENDER Education Customer Lifetime Value Income \\\n", + "1 QZ44356 Arizona F Bachelor 697953.59% 0.0 \n", + "2 AI49188 Nevada F Bachelor 1288743.17% 48767.0 \n", + "17 OE15005 Cali NaN College 394524.16% 28855.0 \n", + "23 TZ98966 Nevada NaN Bachelor 245019.10% 0.0 \n", + "26 US89481 California NaN Bachelor 394637.21% 0.0 \n", + "... ... ... ... ... ... ... \n", + "1059 YG44474 Oregon M College 1401472.13% 54193.0 \n", + "1061 RY92647 Cali F Bachelor 1050677.17% 0.0 \n", + "1068 GS98873 Arizona F Bachelor 323912.47% 16061.0 \n", + "1069 CW49887 California F Master 462680.11% 79487.0 \n", + "1070 MY31220 California F College 899704.02% 54230.0 \n", + "\n", + " Monthly Premium Auto Number of Open Complaints Policy Type \\\n", + "1 94.0 1/0/00 Personal Auto \n", + "2 108.0 1/0/00 Personal Auto \n", + "17 101.0 1/0/00 Personal Auto \n", + "23 73.0 1/3/00 Corporate Auto \n", + "26 111.0 1/0/00 Personal Auto \n", + "... ... ... ... \n", + "1059 117.0 1/0/00 Corporate Auto \n", + "1061 92.0 1/0/00 Personal Auto \n", + "1068 88.0 1/0/00 Personal Auto \n", + "1069 114.0 1/0/00 Special Auto \n", + "1070 112.0 1/0/00 Personal Auto \n", + "\n", + " Vehicle Class Total Claim Amount \n", + "1 Four-Door Car 1131.464935 \n", + "2 Two-Door Car 566.472247 \n", + "17 SUV 647.442031 \n", + "23 Four-Door Car 554.376763 \n", + "26 Four-Door Car 799.200000 \n", + "... ... ... \n", + "1059 SUV 720.752945 \n", + "1061 Four-Door Car 546.524896 \n", + "1068 Four-Door Car 633.600000 \n", + "1069 SUV 547.200000 \n", + "1070 Two-Door Car 537.600000 \n", + "\n", + "[264 rows x 11 columns]\n", + "\n", + "Summary statistics for high claim amounts:\n", + " Income Monthly Premium Auto Total Claim Amount\n", + "count 264.000000 264.000000 264.000000\n", + "mean 23677.344697 165.193182 782.228263\n", + "std 27013.483721 623.930992 292.751640\n", + "min 0.000000 63.000000 537.600000\n", + "25% 0.000000 99.000000 606.521741\n", + "50% 18807.000000 114.000000 679.597985\n", + "75% 42423.750000 133.250000 851.400000\n", + "max 99316.000000 10202.000000 2893.239678\n" + ] + } + ], + "source": [ + "q75 = customer_data_insurance_df['Total Claim Amount'].quantile(0.75)\n", + "\n", + "high_claim_customers = customer_data_insurance_df[customer_data_insurance_df['Total Claim Amount'] > q75]\n", + "\n", + "summary_stats = high_claim_customers.describe()\n", + "\n", + "print(f\"75th Percentile: {q75}\")\n", + "print(\"\\nCustomers with high claim amounts:\")\n", + "print(high_claim_customers)\n", + "print(\"\\nSummary statistics for high claim amounts:\")\n", + "print(summary_stats)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ffa5b31e", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "base", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}