Deep Dive: Coefficient of Variation

The Coefficient of Variation (CV), also known as relative standard deviation, is a standardized measure of dispersion. It’s expressed as a percentage and doesn’t have units. As a result, CV is an excellent measure of variability for comparing data in different scales.

Mathematically, CV is computed as the ratio of the standard deviation to the mean, often multiplied by 100 to get a percentage. The formula is as follows:

Let’s use Numpy’s mean and std function to compute CV in Python.

def calc_cv(data_array) -> float:
  """Calculate coefficient of variation."""
  return np.std(data_array) / np.mean(data_array)

Next, let’s consider another dimensionless measure of dispersion that is QCD.

Scenario 1

Consider the following two datasets showing the monthly sales of a jewelry shop and a bookstore.

• Jewelry shop: The average monthly sales are $10,000 with a standard deviation of $2,000.
• Bookstore: The average monthly sales are $1,000 with a standard deviation of $200.

Let’s generate sample data for both examples using Numpy.

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_theme(context="notebook", style="whitegrid", palette="deep")

np.random.seed(0) # Setting a seed for reproducibility

jewelry_sales = np.random.normal(loc=10000, scale=2000, size=100)  # mean=10000  std=2000
bookstore_sales = np.random.normal(loc=1000, scale=200, size=100)  # mean=1000  std=2000

mean_jewelry, std_jewelry = np.mean(jewelry_sales), np.std(jewelry_sales)
mean_bookstore, std_bookstore = np.mean(bookstore_sales), np.std(bookstore_sales)

cv_jewelry, cv_bookstore = calc_cv(jewelry_sales), calc_cv(bookstore_sales)

print(
  f"Jewelry Shop: \n\t- Mean = ${mean_jewelry:.3f}"
  f"\n\t- Standard Deviation = ${std_jewelry:.3f}"
  f"\n\t- CV = {cv_jewelry:.3f} (dimensionless)"
)
print(
  f"Bookstore: \n\t- Mean = ${mean_bookstore:.3f}"
  f"\n\t- Standard Deviation = ${std_bookstore:.3f}"
  f"\n\t- CV = {cv_bookstore:.3f} (dimensionless)"
)

Jewelry Shop: 
    - Mean = $10119.616
    - Standard Deviation = $2015.764
    - CV = 0.199 (dimensionless)
Bookstore: 
    - Mean = $1016.403
    - Standard Deviation = $206.933
    - CV = 0.204 (dimensionless)

Let’s see the distribution of both datasets and compare their CVs.

Code

fig, ax = plt.subplots()
sns.histplot(jewelry_sales, kde=True, ax=ax, color="blue")
ax.axvline(mean_jewelry, color="red", linestyle="--", label=f"Mean")
ax.axvline(mean_jewelry - std_jewelry, color="green", linestyle="--", label="1 StdDev")
ax.axvline(mean_jewelry + std_jewelry, color="green", linestyle="--")
ax.set_title(f"Histogram of Jewelry Sales \n(mean={mean_jewelry:.2f}, std={std_jewelry:.2f})")
ax.legend()

fig, ax = plt.subplots()
sns.histplot(bookstore_sales, kde=True, ax=ax, color="blue")
ax.axvline(mean_bookstore, color="red", linestyle="--", label=f"Mean")
ax.axvline(mean_bookstore - std_bookstore, color="green", linestyle="--", label="1 StdDev")
ax.axvline(mean_bookstore + std_bookstore, color="green", linestyle="--")
ax.set_title(f"Histogram of Bookstore Sales \n(mean={mean_bookstore:.2f}, std={std_bookstore:.2f})")
ax.legend()

fig, ax = plt.subplots()
sns.barplot(x=['Jewelry', 'Bookstore'], y=[cv_jewelry, cv_bookstore], ax=ax)
ax.set_title("Coefficient of Variation between Jewelry and Bookstore Monthly Sales")
ax.set_ylabel("CV")

plt.show()

The jewelry shop’s average sales and standard deviation are substantially larger than the bookstore’s (mean of $10,119 and standard deviation of $2,015 compared to the mean of $1,016 with standard deviation of $206), yet their CVs are the same (20%).

This means that relative to their respective average sales, both the jewelry shop and the bookstore have the same relative variablity despite their huge differences in sale volumes (and their standard deviation).

This exemplifies the idea of CV as a relative measure of variability and shows how it can be applied to make comparisons between datasets of different scales.

Scenario 2

Consider two datasets of employee ages from two firms.

Let’s say: - Company A (a startup): Younger workers, some elderly. - Company B (a well-established): Older workers, some younger.

Let’s generate sample data for both examples using Numpy.

np.random.seed(42) # Setting the seed for reproducibility

ages_company_A = np.random.normal(loc=25, scale=3, size=50)  # mean = 25 yrs, std = 3 yrs
ages_company_B = np.random.normal(loc=45, scale=5, size=50)  # mean = 45 yrs, std = 5 yrs

# Add a few outliers
ages_company_A = np.append(ages_company_A, [60, 62, 64])
ages_company_B = np.append(ages_company_B, [20, 22, 24])

# Compute Q1, Q3, and IQR, and QCD for both datasets
ages_company_A_q1, ages_company_A_q3 = np.percentile(ages_company_A, [25, 75])
ages_company_A_iqr = ages_company_A_q3 - ages_company_A_q1  # IQR = Q3 - Q1
ages_company_B_q1, ages_company_B_q3 = np.percentile(ages_company_B, [25, 75])
ages_company_B_iqr = ages_company_B_q3 - ages_company_B_q1

ages_company_A_qcd = calc_qcd(ages_company_A)
ages_company_B_qcd = calc_qcd(ages_company_B)

print(
  f"Company A: \n\t- Q1 = {ages_company_A_q1:.3f} years"
  f"\n\t- Q3 = {ages_company_A_q3:.3f} years"
  f"\n\t- IQR = {ages_company_A_iqr:.3f} years"
  f"\n\t- QCD = {ages_company_A_qcd:.3f} (dimensionless)"
)
print(
  f"Company B: \n\t- Q1 = {ages_company_B_q1:.3f} years"
  f"\n\t- Q3 = {ages_company_B_q3:.3f} years"
  f"\n\t- IQR = {ages_company_B_iqr:.3f} years"
  f"\n\t- QCD = {ages_company_B_qcd:.3f} (dimensionless)"
)

Company A: 
    - Q1 = 22.840 years
    - Q3 = 26.490 years
    - IQR = 3.650 years
    - QCD = 0.074 (dimensionless)
Company B: 
    - Q1 = 42.351 years
    - Q3 = 47.566 years
    - IQR = 5.215 years
    - QCD = 0.058 (dimensionless)

Now, let’s plot the distribution of the data along with the boxplot and QCD to visualise the information above.

Code

fig, ax = plt.subplots()
sns.histplot(ages_company_A, ax=ax, color="blue")
ax.set_title(f"Histogram of the Age of Company A Employees \n(mean={np.mean(ages_company_A):.2f}, std={np.std(ages_company_A):.2f})")

fig, ax = plt.subplots()
sns.histplot(ages_company_B, ax=ax, color="blue")
ax.set_title(f"Histogram of the Age of Company B Employees \n(mean={np.mean(ages_company_B):.2f}, std={np.std(ages_company_B):.2f})")

import pandas as pd
df1 = pd.DataFrame(ages_company_A, columns=["age"])
df1["company"] = "Company A"
df2 = pd.DataFrame(ages_company_B, columns=["age"])
df2["company"] = "Company B"

# Create box plots
plt.figure(figsize=(10, 6))
sns.boxplot(x="company", y="age", data=pd.concat([df1, df2]))
plt.title("Boxplots of Test Scores for Classroom A and Classroom B")

fig, ax = plt.subplots()
sns.barplot(x=["Company A", "Company B"], y=[ages_company_A_qcd, ages_company_B_qcd], ax=ax)
ax.set_title("Coefficient of Variation between Jewelry and Bookstore monthly sales")
ax.set_ylabel("QCD")

plt.show()

Company B’s IQR (5.215 years vs. 3.65 years) suggests wider age dispersion. However, Company B’s elderly staff affects this.

On the other hand, Company A has a larger QCD (0.074 vs. 0.058) than Company B, showing a greater age distribution variation relative to size. The IQR doesn’t reveal this.

In the upcoming sections, we’ll learn how to quantify this difference using the Coefficient of Variation and the Quartile Coefficient of Dispersion.

Why not focus on measures like standard deviation or IQR?

We use standard deviation and IQR to quantify dispersion in datasets. The standard deviation shows the average data point distance from the mean. The IQR shows the distribution of the middle 50% of our data.

However, these measures may be deceptive when comparing the dispersion of two or more datasets with different units or scales, skewed distributions, or in the presense of outliers.

While standard deviation and IQR are useful statistical tools, we occasionally require CV and QCD to conduct fair comparisons.

The CV and QCD both measure and compare variability, although they do it in somewhat different ways. Your data and desired variability determine which one to use.

When to use CV?

CV is a good way to compare the amount of variation in different datasets that have different sizes, units, or average values. Because the CV is a relative measure of spread, it shows how different things are from the mean.

The mean and standard deviation, two measures that are greatly affected by “outliers,” are used to create the CV. So, the CV can give a distorted view of spread in datasets that aren’t normally distributed or have outliers. Thus, CV works best with data that is evenly spread out and doesn’t have any extreme values.

In the sales case, the price ranges for these two groups are very different, so the scales used to measure their sales are also very different. The jewelry store is likely to have much higher average sales and much more variation. If we used the standard deviation to measure how variable these two groups are, we might come to the wrong conclusion that the jewelry shop’s sales are more variable.

The CV allowed us to compare the variability of sales between the two datasets, regardless of their different scales. If the CV is higher for one category, it means that the sales are more variable relative to the average sales for that category.

When to use QCD?

The QCD uses dataset quartiles, which are less outlier-sensitive. QCD is a robust dispersion measure for skewed distributions or datasets containing outliers. The QCD concentrates on the center 50% of the data, which may better capture dispersion in such datasets.

In our example, we examined the age differences between two companies: a startup company (A) with mostly younger employees, and a well-established company (B) with mostly elderly. Given their distinct age ranges, the median age and variability would be higher for the older company. Using the Interquartile Range (IQR) to compare dispersion might inaccurately suggest higher age variance in the established company, as IQR measures absolute variability and is higher for larger values.

The QCD is more effective as it standardizes variability against the median, enabling us to compare age variability between companies on different scales. A higher QCD indicates greater age variance relative to the median for that company. Therefore, the QCD was chosen for this comparison as it accounts for different scales and potential data skew or outliers.

Takeaways

Choosing between CV and QCD depends on the nature of your dataset and analysis goals. Below are key points about both measures:

• Coefficient of Variation (CV)
  • CV is calculated as the ratio of the standard deviation to the mean.
  • CV is dimensionless.
  • Higher CV indicates greater variability relative to the mean.
  • CV could give misleading results if the mean is near zero (divising by zero!).
• Quartile Coefficient of Dispersion (QCD)
  • QCD is based on quartiles.
  • QCD is a robust measure (less sensitive to extreme values).
  • QCD is dimensionless.
  • Higher QCD indicates higher variability of values relative to the median.
  • QCD does not fully capture the spread if the distribution’s tails are important.

Installation

You can install the library via pip:

pip install scikit-llm

Configuration

Before you start using Scikit-LLM, you need to pass your OpenAI API key to Scikit-LLM. You can check out this post to set up your OpenAI API key.

from skllm.config import SKLLMConfig
import os

from dotenv import load_dotenv
load_dotenv()

OPENAI_SECRET_KEY = os.environ["OPENAI_SECRET_KEY"]
OPENAI_ORG_ID = os.environ["OPENAI_ORG_ID"]

SKLLMConfig.set_openai_key(OPENAI_SECRET_KEY)
SKLLMConfig.set_openai_org(OPENAI_ORG_ID)

Single label ZeroShotGPTClassifier

Let’s do a sentiment analysis of a few movie reviews. For training purposes, we define the sentiment for each review (defined by a variable movie_review_labels). We train the model with these reviews and labels, so that we can predict new movie reviews using the trained model.

The sample dataset for the movie reviews is given below:

movie_reviews = [
    "This movie was absolutely wonderful. The storyline was compelling and the characters were very realistic.",
    "I really loved the film! The plot had a few unexpected twists which kept me engaged till the end.",
    "The movie was alright. Not great, but not bad either. A decent one-time watch.",
    "I didn't enjoy the film that much. The plot was quite predictable and the characters lacked depth.",
    "This movie was not to my taste. It felt too slow and the storyline wasn't engaging enough.",
    "The film was okay. It was neither impressive nor disappointing. It was just fine.",
    "I was blown away by the movie! The cinematography was excellent and the performances were top-notch.",
    "I didn't like the movie at all. The story was uninteresting and the acting was mediocre at best.",
    "The movie was decent. It had its moments but was not consistently engaging."
]

movie_review_labels = [
    "positive", 
    "positive", 
    "neutral", 
    "negative", 
    "negative", 
    "neutral", 
    "positive", 
    "negative", 
    "neutral"
]

new_movie_reviews = [
    # A positive review
    "The movie was fantastic! I was captivated by the storyline from beginning to end.",

    # A negative review
    "I found the film to be quite boring. The plot moved too slowly and the acting was subpar.",

    # A neutral review
    "The movie was okay. Not the best I've seen, but certainly not the worst."
]

Let’s train the model and then check what the model predicts for each new review.

from skllm import ZeroShotGPTClassifier

# Initialize the classifier with the OpenAI model
clf = ZeroShotGPTClassifier(openai_model="gpt-3.5-turbo")

# Train the model 
clf.fit(X=movie_reviews, y=movie_review_labels)  

# Use the trained classifier to predict the sentiment of the new reviews
predicted_movie_review_labels = clf.predict(X=new_movie_reviews)  

for review, sentiment in zip(new_movie_reviews, predicted_movie_review_labels):
    print(f"Review: {review}\nPredicted Sentiment: {sentiment}\n\n")

  0%|          | 0/3 [00:00<?, ?it/s] 33%|███▎      | 1/3 [00:01<00:02,  1.21s/it] 67%|██████▋   | 2/3 [00:02<00:01,  1.28s/it]100%|██████████| 3/3 [00:03<00:00,  1.23s/it]100%|██████████| 3/3 [00:03<00:00,  1.24s/it]

Review: The movie was fantastic! I was captivated by the storyline from beginning to end.
Predicted Sentiment: positive


Review: I found the film to be quite boring. The plot moved too slowly and the acting was subpar.
Predicted Sentiment: negative


Review: The movie was okay. Not the best I've seen, but certainly not the worst.
Predicted Sentiment: neutral

As can be seen above, the model predicted the sentiment of each movie review correctly.

Multi-Labels ZeroShotGPTClassifier

In the previous section, we had a single-label classifier ([“positive”, “negative”, “neutral”]). Here, we are going to use the MultiLabelZeroShotGPTClassifier estimator to assign multiple labels to a list of restaurant reviews.

restaurant_reviews = [
    "The food was delicious and the service was excellent. A wonderful dining experience!",
    "The restaurant was in a great location, but the food was just average.",
    "The service was very slow and the food was cold when it arrived. Not a good experience.",
    "The restaurant has a beautiful ambiance, and the food was superb.",
    "The food was great, but I found it to be a bit overpriced.",
    "The restaurant was conveniently located, but the service was poor.",
    "The food was not as expected, but the restaurant ambiance was really nice.",
    "Great food and quick service. The location was also very convenient.",
    "The prices were a bit high, but the food quality and the service were excellent.",
    "The restaurant offered a wide variety of dishes. The service was also very quick."
]

restaurant_review_labels = [
    ["Food", "Service"],
    ["Location", "Food"],
    ["Service", "Food"],
    ["Atmosphere", "Food"],
    ["Food", "Price"],
    ["Location", "Service"],
    ["Food", "Atmosphere"],
    ["Food", "Service", "Location"],
    ["Price", "Food", "Service"],
    ["Food Variety", "Service"]
]

new_restaurant_reviews = [
    "The food was excellent and the restaurant was located in the heart of the city.",
    "The service was slow and the food was not worth the price.",
    "The restaurant had a wonderful ambiance, but the variety of dishes was limited."
]

Let’s train the model and then predict the labels for new reviews.

from skllm import MultiLabelZeroShotGPTClassifier

# Initialize the classifier with the OpenAI model
clf = MultiLabelZeroShotGPTClassifier(max_labels=3)

# Train the model 
clf.fit(X=restaurant_reviews, y=restaurant_review_labels)

# Use the trained classifier to predict the labels of the new reviews
predicted_restaurant_review_labels = clf.predict(X=new_restaurant_reviews)

for review, labels in zip(new_restaurant_reviews, predicted_restaurant_review_labels):
    print(f"Review: {review}\nPredicted Labels: {labels}\n\n")

  0%|          | 0/3 [00:00<?, ?it/s] 33%|███▎      | 1/3 [00:01<00:02,  1.44s/it] 67%|██████▋   | 2/3 [00:02<00:01,  1.44s/it]100%|██████████| 3/3 [00:05<00:00,  1.87s/it]100%|██████████| 3/3 [00:05<00:00,  1.76s/it]

Review: The food was excellent and the restaurant was located in the heart of the city.
Predicted Labels: ['Food', 'Location']


Review: The service was slow and the food was not worth the price.
Predicted Labels: ['Service', 'Price']


Review: The restaurant had a wonderful ambiance, but the variety of dishes was limited.
Predicted Labels: ['Atmosphere', 'Food Variety']

The predicted labels for each review are spot-on.

How is an iterator defined in Python?

An iterator object must implement two special methods: __iter__() and __next__(), collectively known as the iterator protocol [2].

The __iter__() method returns the iterator object itself, and is required for your object to be used in any iteration context, such as a for loop. The __next__() method returns the next value from the iterator. If there are no more items to return, it should raise StopIteration.

class CountUpToThree:
    def __init__(self):
        self.count = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.count < 3:
            value = self.count
            self.count += 1
            return value
        else:
            raise StopIteration

counter = CountUpToThree()

for c in counter:
    print(c)

0
1
2

Infinite Iterators

Infinite iterators are a unique feature in the itertools module. They produce an endless sequence of items, only stopping when we explicitly break the loop. This can be particularly useful in scenarios where we have a repeating pattern or want to generate a continuous sequence. However, you must be careful when using these to avoid creating an infinite loop in your program. Let’s look at the three main infinite iterator functions: count(), cycle(), and repeat().

from itertools import count

for idx in count(start=100, step=5):
    print(idx)
    if idx > 110:  # Break the loop to prevent an infinite loop
        break

100
105
110
115

from itertools import cycle

count = 0
for item in cycle("ABC"):
    print(item)
    count += 1
    if count >= 5:  # Break the loop to prevent infinite loop
        break

A
B
C
A
B

from itertools import cycle

items = ["A", "B", "C"]
cycled_items = cycle(items) # an iterator that returns elements from the iterable indefinitely

current_item = next(cycled_items)  # to advance through the iterator

for _ in range(5):
    next_item = next(cycled_items)
    print(f"Current item: {current_item}\nNext item: {next_item}\n")
    current_item = next_item

Current item: A
Next item: B

Current item: B
Next item: C

Current item: C
Next item: A

Current item: A
Next item: B

Current item: B
Next item: C

from itertools import repeat

for i in repeat(["A", "B"], times=3):
    print(i)
print("\n")
for i in repeat("AB", times=3):
    print(i)

['A', 'B']
['A', 'B']
['A', 'B']


AB
AB
AB

Combinatoric Iterators

Combinatoric iterators are used to create different types of iterators that generate all possible combinations, permutations, or Cartesian products (a set of all ordered pairs) of an iterable¹. They are powerful tools when we need to consider all possible combinations of elements. Here we’ll focus on three functions: product(), permutations(), and combinations().

from itertools import product

for item in product(["A", "B"], repeat=2):
    print(item)

('A', 'A')
('A', 'B')
('B', 'A')
('B', 'B')

from itertools import permutations

for item in permutations("ABC", r=2):  # equivalent to permutations(["A", "B", "C"], 2)
    print(item)

('A', 'B')
('A', 'C')
('B', 'A')
('B', 'C')
('C', 'A')
('C', 'B')

from itertools import combinations

for item in combinations(["A", "B", "C"], r=2):
    print(item)

('A', 'B')
('A', 'C')
('B', 'C')

Terminating Iterators

Functions that return a single iterable after using up all elements in the input iterable are called terminating iterators. They are used to reduce the input iterable in some way. For this section, we’ll focus on accumulate(), groupby(), and chain().

from itertools import accumulate

list_ = [3, 4, 6, 2, 1, 9, 8]
for item in accumulate(list_, func=max):
    print(item)
# accumulate([3], func=max) -> 3
# accumulate([3, 4], func=max) -> 4
# accumulate([3, 4, 6], func=max) -> 6
# accumulate([3, 4, 6, 2], func=max) -> 6
# accumulate([3, 4, 6, 2, 1], func=max) -> 6
# accumulate([3, 4, 6, 2, 1, 9], func=max) -> 9
# accumulate([3, 4, 6, 2, 1, 9, 8], func=max) -> 9

3
4
6
6
6
9
9

from itertools import groupby

list_ = [
    ("apple", "fruit"), 
    ("orange", "fruit"), 
    ("lettuce", "vegetable"), 
    ("spinach", "vegetable")
]
for key, group in groupby(list_, key=lambda x: x[1]):
    print(f'"{key}" group: ', list(group))

"fruit" group:  [('apple', 'fruit'), ('orange', 'fruit')]
"vegetable" group:  [('lettuce', 'vegetable'), ('spinach', 'vegetable')]

from itertools import chain

list_1 = ["A", "B"]
list_2 = [1, 2, 3]
s = "cd"

for each in chain(list_1, list_2, s):
    print(each)

A
B
1
2
3
c
d

Installation

You can install the library via pip by running pip install string2string.

Needleman-Wunsch Algorithm for Global Alignment

The Needleman-Wunsch algorithm is a type of dynamic programming algorithm that is often used in bioinformatics to match two DNA or protein sequences, globally.

from string2string.alignment import NeedlemanWunsch

nw = NeedlemanWunsch()

# Define two sequences
s1 = 'ACGTGGA'
s2 = 'AGCTCGC'

aligned_s1, aligned_s2, score_matrix = nw.get_alignment(s1, s2, return_score_matrix=True)

print(f'The alignment between "{s1}" and "{s2}":')
nw.print_alignment(aligned_s1, aligned_s2)

The alignment between "ACGTGGA" and "AGCTCGC":
A | C | G | - | T | G | G | A
A | - | G | C | T | C | G | C

For a more informative comparison, we can use plot_pairwise_alignment() function in the library.

from string2string.misc.plotting_functions import plot_pairwise_alignment

path, s1_pieces, s2_pieces = nw.get_alignment_strings_and_indices(aligned_s1, aligned_s2)

plot_pairwise_alignment(
    seq1_pieces=s1_pieces,
    seq2_pieces=s2_pieces,
    alignment=path,
    str2colordict={
        "A": "pink", 
        "G": "lightblue", 
        "C": "lightgreen", 
        "T": "yellow", 
        "-": "lightgray"
    },
    title="",
    seq1_name="Sequence 1",
    seq2_name="Sequence 2",
)

Dynamic Time Warping

DTW is a useful tool to compare two time series that might differ in speed, duration, or both. It discovers the path across these distances that minimizes the total difference between the sequences by calculating the “distance” between each pair of points in the two sequences.

Let’s go over an example using the alignment module in the string2string library.

from string2string.alignment import DTW

dtw = DTW()

x = [3, 1, 2, 2, 1]
y = [2, 0, 0, 3, 3, 1, 0]

dtw_path = dtw.get_alignment_path(x, y, distance="square_difference")
print(f"DTW path: {dtw_path}")

DTW path: [(0, 0), (1, 1), (1, 2), (2, 3), (3, 4), (4, 5), (4, 6)]

Above is an example borrowed from my previous post, An Illustrative Introduction to Dynamic Time Warping. For those looking to delve deeper into the topic, in [2], I explained the core concepts of DTW in a visual and accessible way.

Lexical Search (exact-match search)

Lexical search, in layman’s terms, is the act of searching for certain words or phrases inside a text, analogous to searching for a word or phrase in a dictionary or a book.

Instead of trying to figure out what a string of letters or words means, it just tries to match them exactly. When it comes to search engines and information retrieval, lexical search is a basic strategy to finding relevant resources based on the keywords or phrases users enter, without any attempt at comprehending the linguistic context of the words or phrases in question.

Currently, the string2string library provides the following lexical search algorithm:

• Naive (brute-force) search algorithm
• Rabin-Karp search algorithm
• Knuth-Morris-Pratt (KMP) search algorithm (see the example below)
• Boyer-Moore search algorithm

from string2string.search import KMPSearch

kmp_search = KMPSearch()

pattern = "Redwood tree"
text = "The gentle fluttering of a Monarch butterfly, the towering majesty of a Redwood tree, and the crashing of ocean waves are all wonders of nature."

idx = kmp_search.search(pattern=pattern, text=text)

print(f"The starting index of pattern: {idx}")
print(f'The pattern (± characters) inside the text: "{text[idx-5: idx+len(pattern)+5]}"')

The starting index of pattern: 72
The pattern (± characters) inside the text: "of a Redwood tree, and"

Semantic Search

Semantic search is a more sophisticated method of information retrieval that goes beyond simple word or phrase searches. It employs NLP (natural language processing) to decipher a user’s intent and return accurate results.

To put it another way, let’s say you’re interested in “how to grow apples.” While a lexical search may produce results including the terms “grow” and “apples,” a semantic search will recognize that you are interested in the cultivation of apple trees and deliver results accordingly. The search engine would then prioritize results that not only included the phrases it was looking for but also gave relevant information about planting, trimming, and harvesting apple trees.

corpus = {"text": [
    "A warm cup of tea in the morning helps me start the day right.",
    "Staying active is important for maintaining a healthy lifestyle.",
    "I find inspiration in trying out new activities or hobbies.",
    "The view from my window is always a source of inspiration.",
    "The encouragement from my loved ones keeps me going.",
    "The novel I've picked up recently has been a page-turner.",
    "Listening to podcasts helps me stay focused during work.",
    "I can't wait to explore the new art gallery downtown.",
    "Meditating in a peaceful environment brings clarity to my thoughts.",
    "I believe empathy is a crucial quality to possess.",
    "I like to exercise a few times a week."
    ]
}

query = "I enjoy walking early morning before I start my work."

from string2string.search import FaissSearch

faiss_search = FaissSearch(
    model_name_or_path = "facebook/bart-large",
    tokenizer_name_or_path = "facebook/bart-large",
)

faiss_search.initialize_corpus(
    corpus=corpus,
    section="text", 
    embedding_type="mean_pooling",
)

top_k_similar_answers = 3
most_similar_results = faiss_search.search(
    query=query,
    k=top_k_similar_answers,
)
    
print(f"Query: {query}\n")
for i in range(top_k_similar_answers):
    print(f'Result {i+1} (score={most_similar_results["score"][i]:.2f}): "{most_similar_results["text"][i]}"')

Query: I enjoy walking early morning before I start my work.

Result 1 (score=208.49): "I find inspiration in trying out new activities or hobbies."
Result 2 (score=218.21): "I like to exercise a few times a week."
Result 3 (score=225.96): "I can't wait to explore the new art gallery downtown."

Levenshtein edit distance

Levenshtein edit distance, or simply the edit distance, is the minimal number of insertions, deletions, or substitutions needed to convert one string into another.

from string2string.distance import LevenshteinEditDistance

edit_dist = LevenshteinEditDistance()

# Create two strings
s1 = "The beautiful cherry blossoms bloom in the spring time."
s2 = "The beutiful cherry blosoms bloom in the spring time."

# Let's compute the edit distance between the two strings and measure the computation time
edit_dist_score  = edit_dist.compute(s1, s2, method="dynamic-programming")

print(f'The distance between the following two sentences is {edit_dist_score}:') #\n- "{s1}"\n- "{s2}"')
print(f'"The beautiful cherry blossoms bloom in the spring time."')
print(f'"The beutiful cherry blosoms bloom in the spring time."')

The distance between the following two sentences is 2.0:
"The beautiful cherry blossoms bloom in the spring time."
"The beutiful cherry blosoms bloom in the spring time."

Jaccard Index

The Jaccard index can be used to quantify the similarity between sets of words or tokens and is commonly used in tasks such as document similarity or topic modeling. For example, the Jaccard index can be used to measure the overlap between the sets of words in two different documents or to identify the most similar topics across a collection of documents.

from string2string.distance import JaccardIndex 

jaccard_dist = JaccardIndex()

# Suppose we have two documents
doc1 = ["red", "green", "blue", "yellow", "purple", "pink"]
doc2 = ["green", "orange", "cyan", "red"]

jaccard_dist_score = jaccard_dist.compute(doc1, doc2)

print(f"Jaccard distance between doc1 and doc2: {jaccard_dist_score:.2f}")

Jaccard distance between doc1 and doc2: 0.75

Example 1

In this part, I’ll use an example from [2] to help you understand why differential privacy is important. In a study that looks at the link between social class and health results, researchers ask subjects for private information like where they live, how much money they have, and their medical background.

John, one of the participants, is worried that his personal information could get out and hurt his applications for life insurance or a mortgage. To make sure that John’s worries are taken care of, the researchers can use differential privacy. This makes sure that any data that is shared won’t reveal specific information about him. Different levels of privacy can be shown by John’s “opt-out” situation, in which his data is left out of the study. This protects his anonymity because the analysis’ results are not tied to any of his personal details.

Differential privacy seeks to protect privacy in the real world as if the data were being looked at in an opt-out situation. Since John’s data is not part of the computation, the results regarding him can only be as accurate as the data available to everyone else.

A precise description of differential privacy requires formal mathematical language and technical concepts, but the basic concept is to protect the privacy of individuals by limiting the information that can be obtained about them from the released data, thereby ensuring that their sensitive information remains private.

Example 2

The U.S. Census Bureau used a differential privacy framework as a part of its disclosure avoidance strategy to strike a compromise between the data collection and reporting needs and the privacy concerns of the respondents. Check [3] to find more information about the confidentiality protection provided by the U.S. Census Bureau. Moreover, [4] provides an explanation of how DP was utilized in the 2020 US Census data here.

The meanining of “Differential” within the realm of DP?

The term “differential” privacy refers to its emphasis on the dissimilarity between the results produced by a privacy-preserving algorithm on two datasets that differ by just one individual’s data.

Mechanism

A mechanism is a mathematical method or process that is used on the data to make sure privacy is maintained while still giving useful information.

Epsilon ()

is a privacy parameter that controls the level of privacy given by a differentially private mechanism. In other words, regulates how much the output of the mechanism can vary between two neighboring databases and measures how much privacy is lost when the mechanism is run on the database [5].

Stronger privacy guarantees are provided by a smaller , but the output may be less useful as a result [6]. controls the amount of noise added to the data and shows how much the output probability distribution can change when the data of a single person is altered.

Delta ()

is an extra privacy option that lets you set how likely it is that your privacy will be compromised. Hence, controls the probability of an extreme privacy breach, where the added noise (controlled by ) does not provide sufficient protection.

is a non-negative number that measures the chance of a data breach. It is usually very small and close to zero. This change makes it easier to do more complicated studies and machine learning models while still protecting privacy see [6].

If is low, there is less of a chance that someone’s privacy is going to get compromised. But this comes at a cost. If is too small, more noise might be introduced into the data, diminishing the quality of the end-result. is one parameter to consider, but it must be balanced with epsilon and the data’s practicality.

Unveiling the Mathematics behind Differential Privacy

Consider two databases, and , that differ by only one record.

Formally, a mechanism is -differentially private if, for any two adjacent datasets and , and for any possible output , the following holds:

However, we can reframe the Equation 1 in terms of divergences, resulting in Equation 2.

Here denotes the Rényi divergence.¹

Post-processing immunity

The differentially private output can be subjected to any function or analysis, and the outcome will continue to uphold the original privacy assurances. For instance, if you apply a differentially private mechanism to a dataset and then take the average age of the individuals in the dataset, the resulting average age will still be differentially private and will provide the same level of privacy assurances as the output it was originally designed to provide.

Thanks to the post-processing feature, we can use differentially private mechanisms in the same way as generic ones. Hence, it is possible to combine several differentially private mechanisms without sacrificing the integrity of differential privacy.

Composition

When multiple differentially private techniques are used on the same data or when queries are combined, composition is the property that ensures the privacy guarantees of differential privacy still apply. Composition can be either sequential or parallel. If you apply two mechanisms, with -DP and with -DP on a dataset, then the composition of and is at least -DP.

Robustness to auxiliary information:

Differential privacy is resistant to auxiliary information attackers, which means that even if an attacker has access to other relevant data, they will not be able to learn anything about a person from a DP output. For instance, if a hospital were to share differentially private information regarding individuals’ medical situations, an attacker with access to other medical records would not be able to greatly increase their knowledge of a given patient from the published numbers.

What K in K-NN stands for?

K in K-Nearest Neighbors refers to the number of neighbors that one should take into consideration when predicting (voting for) the class of a new point. It will get more clear from the below example.

An Illustration of K-NN

As I mentioned earlier, KNN is a supervised learning technique, so we should have a labeled dataset. Let’s say we have two classes as can be seen in below image: Class A (blue points) and Class B (green points). A new data point (red) is given to us and we want to predict whether the new point belongs to Class A or Class B.

Let’s first try K = 3. In this case, we have to find the three closest data points (aka three nearest neighbors) to the new (red) data point. As can be seen from the left side, two of three closest neighbors belong to Class B (green) and one belongs to Class A (blue). So, we should assign the new point to Class B.

Now let’s set K = 5 (right side of above image). In this case, three out of the closest five points belong to Class A, so the new point should be classified as Class A. Unfortunately, there is no specific way of determining K, so we have to try a few values. Very low values of K like 1 or 2 may make the model very complex and sensitive to outliers. A common value for K is 5 see [1].

Pros and Cons

Following are the advantages and drawbacks of KNN see [3]:

Pros

• Useful for nonlinear data because KNN is a nonparametric algorithm.
• Can be used for both classification and regression problems, even though mostly used for classification.

Cons

• Difficult to choose K since there is no statistical way to determine that.
• Slow prediction for large datasets.
• Computationally expensive since it has to store all the training data (Lazy Learner).
• Sensitive to non-normalized dataset.
• Sensitive to presence of irrelevant features.

How to find the best K?

There are several ways to determine K in the K-Means clustering algorithm:

• Elbow Method: A common way to determine the number of ideal cluster (K) in K-means. In this approach, we run the K-means with several candidates and calculate the WCSS. The best K is selected based on a trade-off between the model complexity (overfitting) and the WCSS.
• Silhouette Score: A score between -1 and 1 measuring the similarity among points of a cluster and comparing that with other clusters. A score of -1 indicates that a point is in the wrong cluster, whereas a score of 1 indicates that the point is in the right cluster see [4].
• gap statistics: A method to estimate the number of clusters in a dataset. gap statistic compares the change in the within-cluster variation of output of any clustering technique with an expected reference null distribution see [5].

We usually normalize/standardize continuous variables in the data preprocessing stage in order to avoid variables with much larger values dominating any modeling or analysis process see [6].

Pros and Cons

Some pros and cons of K-Means are given below.

Pros

• High scalability since most of calculations can be run in parallel.

Cons

• The outliers can skew the centroids of clusters.
• Poor performance in higher dimensional.

Considerations before automation

Your script will most likely be different. However, here the following tips may help you get started:

First, run the script on your system. Once you’re happy with the result and you want to automate the workflow, then use the instructions below to setup a scheduled workflow using GitHub Actions.

When developing an automation task, it is always good to think about how to run it starting from a fresh OS! It is as if you have a new system and you try to run your script there. A few question to ask yourself:

• Where should I start?
• What are software/libraries I need to install before running the script?
• Where can I find the script I want to run?
• Do I need to pass some environment variables or sensitive information like passwords?
  • How should I pass sensitive information like passwords or tokens?

I will answer above questions for my workflow. Hopefully this will give you enough information to automate your task!

Cron job examples

Below examples covers different aspects of a cron syntax and all valid characters (* , - /).

• You can specify your schedule by choosing a valid number for each part. * means “every” (* * * * * means at every minute on every hour of every day of the month in every month at every day of the week 🙂). Another example is 30 13 1 * * meaning at 13:30 on day 1 of the month.
• You can have multiple parameters for a given section by using the value list separator ,. For instance, * * * * 0,3 means every minute only on Sunday and Wednesday.
• You can have step values by using /. For instance, /10 * * * * means every 10 minutes.
• You can have a range of values by using dash -. For instance, 4-5 1-10 1 * means every minute between 04:00 - 05:59 AM between day 1 and day 10 of January.

And of course, you can have a combination of above options. For example, */30 1-5 * 1,6 0,1 means every 30 minutes between 01:00-05:59 AM only on Sunday and Monday in January and June.

Check crontab or crontab guru to come up with the cron syntax for your schedule.

Where should we start from?

We can start from a fresh Ubuntu system. So, we have the section below the jobs specifying runs-on: ubuntu-latest that will configures the job to run on a fresh virtual machine containing the latest version of an Ubuntu Linux.

Next step is to clone the current repo. You can achieve this by using uses keyword allowing us to use any action from the GitHub Actions Marketplace . We can use the master branch of actions/checkout here (you can also specify the version like actions/checkout@v2).

- name: 🍽️ Checkout the repo
  uses: actions/checkout@master
  with:
    fetch-depth: 1

Which software/libraries we must install?

This step is only necessary if you have to install a library. In my case, I have to first install Python 3.8. This can be achieved by using the actions/setup-python@v2 GitHub Action. Afterwards, we want to install the python package. We can install the Zotero2Readwise package by running pip install zotero2readwise. However, in order to execute a command on the runner, we have to use the run keyword.

- name: 🐍 Set up Python 3.8
  uses: actions/setup-python@v2
  with:
    python-version: '3.8'

- name: 💿 Install Zotero2Readwise Python package
  run: pip install zotero2readwise

Where can I find the script I want to run?

If the script you are trying to run lives in the same repository, you can just skip this step. But here, since the Python script I want to run lives in another GitHub repository, I have to download the script using the curl Linux command.

- name: 📥 Download the Python script needed for automation
  run:  curl https://raw.githubusercontent.com/e-alizadeh/Zotero2Readwise/master/zotero2readwise/run.py -o run.py

Run the script

Now that we have set up our environment, we can run the script as mentioned earlier in the Requirements section.

But one last point is that since we need to pass some sensitive information (like tokens), we can achieve that by passing the secrets to Settings → Secrets → New repository secret.

These secrets will then be available using the following syntax: ${{ secrets.YOUR_SECRET_NAME }} in your YAML file.

For more information about handling variables and secrets, you can check the following two pages on the GitHub Docs about Environment variables and Encrypted secrets.

Now that we have added our secrets, we can run the script as following:

- name: 🚀 Run Automation
  run: python run.py ${{ secrets.READWISE_TOKEN }} ${{ secrets.ZOTERO_KEY }} ${{ secrets.ZOTERO_ID }}

Putting everything together

The file containing all steps above is shown below. The file lives on GitHub.

name: Zotero to Readwise Automation

on:
  push:
    branches:
      - master
  schedule:
    - cron: "0 3 * * 1,3,5" # Runs at 03:00 AM (UTC) every Monday, Wednesday, and Friday (Check https://crontab.guru/)

jobs:
  zotero-to-readwise-automation:
    runs-on: ubuntu-latest
    steps:
      - name: 🍽️ Checkout the repo
        uses: actions/checkout@master
        with:
          fetch-depth: 1

      - name: 🐍 Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'

      - name: 💿 Install Zotero2Readwise Python package
        run: pip install zotero2readwise

      - name: 📥 Download the Python script needed for automation
        run:  curl https://raw.githubusercontent.com/e-alizadeh/Zotero2Readwise/master/zotero2readwise/run.py -o run.py

      - name: 🚀 Run Automation
        run: python run.py ${{ secrets.READWISE_TOKEN }} ${{ secrets.ZOTERO_KEY }} ${{ secrets.ZOTERO_ID }}

PandasTutor Creators

Pandas Tutor was created by Sam Lau and Philip Guo at UC San Diego. This tool is mainly developed for teaching purposes as its creator stated here. This explains some of the limitations this tool have (I will cover some of those limitations later in the post).

A similar tool called Tidy Data Tutor but for R users is created by Sean Kross and Philip Guo.

Dataset

Let’s use the Heart Failure Prediction Dataset Kaggle Dataset (available here). The data is available under Open Database (ODbl) License allowing “users to freely share, modify, and use this Database while maintaining this same freedom for others.” Since Pandas Tutor only works with small data, I will take the first 50 rows of hearts data).

Code

Below is the code used for the visualization in this post. You may notice that the CSV data is encoded here which is a current limitation of this tool.

import pandas as pd
import io

csv = '''
Age,Sex,ChestPainType,RestingBP,Cholesterol,FastingBS,RestingECG,MaxHR,ExerciseAngina,Oldpeak,ST_Slope,HeartDisease
40,M,ATA,140,289,0,Normal,172,N,0,Up,0
49,F,NAP,160,180,0,Normal,156,N,1,Flat,1
37,M,ATA,130,283,0,ST,98,N,0,Up,0
48,F,ASY,138,214,0,Normal,108,Y,1.5,Flat,1
54,M,NAP,150,195,0,Normal,122,N,0,Up,0
39,M,NAP,120,339,0,Normal,170,N,0,Up,0
45,F,ATA,130,237,0,Normal,170,N,0,Up,0
54,M,ATA,110,208,0,Normal,142,N,0,Up,0
37,M,ASY,140,207,0,Normal,130,Y,1.5,Flat,1
48,F,ATA,120,284,0,Normal,120,N,0,Up,0
37,F,NAP,130,211,0,Normal,142,N,0,Up,0
58,M,ATA,136,164,0,ST,99,Y,2,Flat,1
39,M,ATA,120,204,0,Normal,145,N,0,Up,0
49,M,ASY,140,234,0,Normal,140,Y,1,Flat,1
42,F,NAP,115,211,0,ST,137,N,0,Up,0
54,F,ATA,120,273,0,Normal,150,N,1.5,Flat,0
38,M,ASY,110,196,0,Normal,166,N,0,Flat,1
43,F,ATA,120,201,0,Normal,165,N,0,Up,0
60,M,ASY,100,248,0,Normal,125,N,1,Flat,1
36,M,ATA,120,267,0,Normal,160,N,3,Flat,1
43,F,TA,100,223,0,Normal,142,N,0,Up,0
44,M,ATA,120,184,0,Normal,142,N,1,Flat,0
49,F,ATA,124,201,0,Normal,164,N,0,Up,0
44,M,ATA,150,288,0,Normal,150,Y,3,Flat,1
40,M,NAP,130,215,0,Normal,138,N,0,Up,0
36,M,NAP,130,209,0,Normal,178,N,0,Up,0
53,M,ASY,124,260,0,ST,112,Y,3,Flat,0
52,M,ATA,120,284,0,Normal,118,N,0,Up,0
53,F,ATA,113,468,0,Normal,127,N,0,Up,0
51,M,ATA,125,188,0,Normal,145,N,0,Up,0
53,M,NAP,145,518,0,Normal,130,N,0,Flat,1
56,M,NAP,130,167,0,Normal,114,N,0,Up,0
54,M,ASY,125,224,0,Normal,122,N,2,Flat,1
41,M,ASY,130,172,0,ST,130,N,2,Flat,1
43,F,ATA,150,186,0,Normal,154,N,0,Up,0
32,M,ATA,125,254,0,Normal,155,N,0,Up,0
65,M,ASY,140,306,1,Normal,87,Y,1.5,Flat,1
41,F,ATA,110,250,0,ST,142,N,0,Up,0
48,F,ATA,120,177,1,ST,148,N,0,Up,0
48,F,ASY,150,227,0,Normal,130,Y,1,Flat,0
54,F,ATA,150,230,0,Normal,130,N,0,Up,0
54,F,NAP,130,294,0,ST,100,Y,0,Flat,1
35,M,ATA,150,264,0,Normal,168,N,0,Up,0
52,M,NAP,140,259,0,ST,170,N,0,Up,0
43,M,ASY,120,175,0,Normal,120,Y,1,Flat,1
59,M,NAP,130,318,0,Normal,120,Y,1,Flat,0
37,M,ASY,120,223,0,Normal,168,N,0,Up,0
50,M,ATA,140,216,0,Normal,170,N,0,Up,0
36,M,NAP,112,340,0,Normal,184,N,1,Flat,0
41,M,ASY,110,289,0,Normal,170,N,0,Flat,1
'''

df_hearts = pd.read_csv(io.StringIO(csv))
df_hearts = df_hearts[
    ["Age", "Sex", "RestingBP", "ChestPainType", "Cholesterol", "HeartDisease"]
]

(df_hearts.sort_values("Age")
.groupby(["Sex", "HeartDisease"])
.agg({"RestingBP": ["mean", "std"], 
      "Cholesterol": ["mean", "std"],
      "Sex": ["count"]
      })
)

So our transformations is only the last few lines

(df_hearts.sort_values("Age")
.groupby(["Sex", "HeartDisease"])
.agg({"RestingBP": ["mean", "std"], 
      "Cholesterol": ["mean", "std"],
      "Sex": ["count"]
      })
)

Results

https://pandastutor.com/vis.html#code=import%20pandas%20as%20pd%0Aimport%20io%0A%0Acsv%20%3D%20'''%0AAge,Sex,ChestPainType,RestingBP,Cholesterol,FastingBS,RestingECG,MaxHR,ExerciseAngina,Oldpeak,ST_Slope,HeartDisease%0A40,M,ATA,140,289,0,Normal,172,N,0,Up,0%0A49,F,NAP,160,180,0,Normal,156,N,1,Flat,1%0A37,M,ATA,130,283,0,ST,98,N,0,Up,0%0A48,F,ASY,138,214,0,Normal,108,Y,1.5,Flat,1%0A54,M,NAP,150,195,0,Normal,122,N,0,Up,0%0A39,M,NAP,120,339,0,Normal,170,N,0,Up,0%0A45,F,ATA,130,237,0,Normal,170,N,0,Up,0%0A54,M,ATA,110,208,0,Normal,142,N,0,Up,0%0A37,M,ASY,140,207,0,Normal,130,Y,1.5,Flat,1%0A48,F,ATA,120,284,0,Normal,120,N,0,Up,0%0A37,F,NAP,130,211,0,Normal,142,N,0,Up,0%0A58,M,ATA,136,164,0,ST,99,Y,2,Flat,1%0A39,M,ATA,120,204,0,Normal,145,N,0,Up,0%0A49,M,ASY,140,234,0,Normal,140,Y,1,Flat,1%0A42,F,NAP,115,211,0,ST,137,N,0,Up,0%0A54,F,ATA,120,273,0,Normal,150,N,1.5,Flat,0%0A38,M,ASY,110,196,0,Normal,166,N,0,Flat,1%0A43,F,ATA,120,201,0,Normal,165,N,0,Up,0%0A60,M,ASY,100,248,0,Normal,125,N,1,Flat,1%0A36,M,ATA,120,267,0,Normal,160,N,3,Flat,1%0A43,F,TA,100,223,0,Normal,142,N,0,Up,0%0A44,M,ATA,120,184,0,Normal,142,N,1,Flat,0%0A49,F,ATA,124,201,0,Normal,164,N,0,Up,0%0A44,M,ATA,150,288,0,Normal,150,Y,3,Flat,1%0A40,M,NAP,130,215,0,Normal,138,N,0,Up,0%0A36,M,NAP,130,209,0,Normal,178,N,0,Up,0%0A53,M,ASY,124,260,0,ST,112,Y,3,Flat,0%0A52,M,ATA,120,284,0,Normal,118,N,0,Up,0%0A53,F,ATA,113,468,0,Normal,127,N,0,Up,0%0A51,M,ATA,125,188,0,Normal,145,N,0,Up,0%0A53,M,NAP,145,518,0,Normal,130,N,0,Flat,1%0A56,M,NAP,130,167,0,Normal,114,N,0,Up,0%0A54,M,ASY,125,224,0,Normal,122,N,2,Flat,1%0A41,M,ASY,130,172,0,ST,130,N,2,Flat,1%0A43,F,ATA,150,186,0,Normal,154,N,0,Up,0%0A32,M,ATA,125,254,0,Normal,155,N,0,Up,0%0A65,M,ASY,140,306,1,Normal,87,Y,1.5,Flat,1%0A41,F,ATA,110,250,0,ST,142,N,0,Up,0%0A48,F,ATA,120,177,1,ST,148,N,0,Up,0%0A48,F,ASY,150,227,0,Normal,130,Y,1,Flat,0%0A54,F,ATA,150,230,0,Normal,130,N,0,Up,0%0A54,F,NAP,130,294,0,ST,100,Y,0,Flat,1%0A35,M,ATA,150,264,0,Normal,168,N,0,Up,0%0A52,M,NAP,140,259,0,ST,170,N,0,Up,0%0A43,M,ASY,120,175,0,Normal,120,Y,1,Flat,1%0A59,M,NAP,130,318,0,Normal,120,Y,1,Flat,0%0A37,M,ASY,120,223,0,Normal,168,N,0,Up,0%0A50,M,ATA,140,216,0,Normal,170,N,0,Up,0%0A36,M,NAP,112,340,0,Normal,184,N,1,Flat,0%0A41,M,ASY,110,289,0,Normal,170,N,0,Flat,1%0A'''%0A%0Adf_hearts%20%3D%20pd.read_csv%28io.StringIO%28csv%29%29%0Adf_hearts%20%3D%20df_hearts%5B%0A%20%20%20%20%5B%22Age%22,%20%22Sex%22,%20%22RestingBP%22,%20%22ChestPainType%22,%20%22Cholesterol%22,%20%22HeartDisease%22%5D%0A%5D%0A%0A%28df_hearts.sort_values%28%22Age%22%29%0A.groupby%28%5B%22Sex%22,%20%22HeartDisease%22%5D%29%0A.agg%28%7B%22RestingBP%22%3A%20%5B%22mean%22,%20%22std%22%5D,%20%0A%20%20%20%20%20%20%22Cholesterol%22%3A%20%5B%22mean%22,%20%22std%22%5D,%0A%20%20%20%20%20%20%22Sex%22%3A%20%5B%22count%22%5D%0A%20%20%20%20%20%20%7D%29%0A%29&d=2021-12-08&lang=py&v=v1

Pros:

• Step-by-step visualization
• Interactive plots (you can track the data rows before and after the transformation)
• Shareable URL

Cons (current limitations):

• Only works for small codes (The code should be 5000bytes). Since the data is also encoded and not read from a file, hence, you can only visualize small datasets.
• As stated in the previous step, you have to encode the data along with the code as reading from external resources (files or links) are not supported.
• Limited Pandas’ methods support.
• You can visualize the Pandas expression only on the last line. You may have to pipe multiple steps together or run the visualizations separately.

For a complete list of unsupported features or other FAQ, you can check here.

Create Counterfactual (for model interpretability)

For creating counterfactual records (in the context of machine learning), we need to modify the features of some records from the training set in order to change the model predictionsee [2]. This may be helpful in explaining the behavior of a trained model. The algorithm used in the library to create counterfactual records is developed by Wachter et al see [3].

You can create counterfactual records using create_counterfactual() from the library. Note that this implementation works with any scikit-learn estimator that supports the predict() function. Below is an example of creating a counterfactual record for an ML model. The counterfactual record is highlighted in a red dot within the classifier’s decision regions (we will go over how to draw decision regions of classifiers later in the post).

from sklearn.linear_model import LogisticRegression
clf_logistic_regression = LogisticRegression(random_state=0)
clf_logistic_regression.fit(X_2d, y)

from mlxtend.evaluate import create_counterfactual
from mlxtend.plotting import plot_decision_regions

counterfact = create_counterfactual(
    x_reference=X_2d[15], 
    y_desired=2, # Desired class
    model=clf_logistic_regression, 
    X_dataset=X_2d,
    y_desired_proba=0.95,
    lammbda=1, 
    random_seed=123
)

scatter_highlight_defaults = {
    'c': 'red',
    'edgecolor': 'yellow',
    'alpha': 1.0,
    'linewidths': 2,
    'marker': 'o',
    's': 120
}

fig, ax = plt.subplots(figsize=(10, 6))
plot_decision_regions(X_2d, y, clf=clf_logistic_regression, legend=2, ax=ax)

ax.tick_params(axis='both', which='major', labelsize=24)
ax.set_title("Create a Counterfactual Record", fontsize=24, fontweight="bold")
ax.set_xlabel("Color.int", fontsize=20, fontweight="bold")
ax.set_ylabel("Phenols", fontsize=20, fontweight="bold")

ax.scatter(
    *counterfact,
    **scatter_highlight_defaults
)

PCA Correlation Circle

An interesting and different way to look at PCA results is through a correlation circle that can be plotted using plot_pca_correlation_graph(). We basically compute the correlation between the original dataset columns and the PCs (principal components). Then, these correlations are plotted as vectors on a unit-circle. The axes of the circle are the selected dimensions (a.k.a. PCs). You can specify the PCs you’re interested in by passing them as a tuple to dimensions function argument. The correlation circle axes labels show the percentage of the explained variance for the corresponding PCsee [1].

Remember that the normalization is important in PCA because the PCA projects the original data on to the directions that maximize the variance.

from mlxtend.plotting import plot_pca_correlation_graph
from sklearn.preprocessing import StandardScaler

X_norm = StandardScaler().fit_transform(X) # Normalizing the feature columns is recommended (X - mean) / std

fig, correlation_matrix = plot_pca_correlation_graph(
    X_norm, 
    attribute_names,
    dimensions=(1, 2),
    figure_axis_size=6
)

PCA correlation circle diagram between the first two principal components and all data attributes

Correlation matrix between wine features and the first two PCs

Bias-Variance Decomposition

You often hear about the bias-variance tradeoff to show the model performance. In supervised learning, the goal often is to minimize both the bias error (to prevent underfitting) and variance (to prevent overfitting) so that our model can generalize beyond the training set see [4]. This process is known as a bias-variance tradeoff.

Note that we cannot calculate the actual bias and variance for a predictive model, and the bias-variance tradeoff is a concept that an ML engineer should always consider and tries to find a sweet spot between the two.Having said that, we can still study the model’s expected generalization error for certain problems. In particular, we can use the bias-variance decomposition to decompose the generalization error into a sum of 1) bias, 2) variance, and 3) irreducible error[4,5].

The bias-variance decomposition can be implemented through bias_variance_decomp() in the library. An example of such implementation for a decision tree classifier is given below.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X_df.values, y,
                                                    test_size=0.3,
                                                    random_state=123,
                                                    shuffle=True,
                                                    stratify=y)
tree = DecisionTreeClassifier(random_state=123)

from mlxtend.evaluate import bias_variance_decomp

avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(
        tree, X_train, y_train, X_test, y_test, 
        loss='mse',
        num_rounds=50, # Number of bootstrap rounds for implementing the decomposition
        random_seed=123
)

print(f"Average expected loss: {avg_expected_loss.round(3)}")
print(f"Average bias: {avg_bias.round(3)}")
print(f"Average variance: {avg_var.round(3)}")

>>> Average expected loss: 0.108 
>>> Average bias: 0.032 
>>> Average variance: 0.076

Plotting Decision Regions of Classifiers

MLxtend library has an out-of-the-box function plot_decision_regions() to draw a classifier’s decision regions in 1 or 2 dimensions.

Here, I will draw decision regions for several scikit-learn as well as MLxtend models. Let’s first import the models and initialize them.

# Models
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB 
from mlxtend.classifier import EnsembleVoteClassifier 

# Initializing Classifiers
clf_logistic_regression = LogisticRegression(random_state=0)
clf_nb = GaussianNB()
clf_random_forest = RandomForestClassifier(random_state=0)
clf_ensemble = EnsembleVoteClassifier(
    clfs=[clf_logistic_regression, clf_nb, clf_random_forest], 
    weights=[2, 1, 1], 
    voting='soft'
)

all_classifiers = [
    ("Logistic Regression", clf_logistic_regression),
    ("Naive Bayes", clf_nb),
    ("Random Forest", clf_random_forest),
    ("Ensemble", clf_ensemble),
]

Now that we have initialized all the classifiers, let’s train the models and draw decision boundaries using plot_decision_regions() from the MLxtend library.

from mlxtend.plotting import plot_decision_regions
from itertools import product  # Used to generate indices for figure subplots!

fig, axs = plt.subplots(2, 2, figsize=(28, 24), sharey=True)

for classifier, grid in zip(
    all_classifiers,
    product([0, 1], [0, 1])  # generate [(0, 0), (0, 1), (1, 0), (1, 1)]
):
    clf_name, clf = classifier[0], classifier[1]
    ax = axs[grid[0], grid[1]]

    clf.fit(X_2d, y)
    
    plot_decision_regions(
        X=X_2d, 
        y=y, 
        clf=clf, 
        legend=2, 
        ax=ax
    )

    ax.set_title(clf_name, fontsize=24, fontweight="bold")
    ax.tick_params(axis='both', which='major', labelsize=18)
    ax.set_xlabel("Color.int", fontsize=20, fontweight="bold")
    ax.set_ylabel("Phenols", fontsize=20, fontweight="bold")

Matrix of Scatter Plots

Another useful tool from MLxtend is the ability to draw a matrix of scatter plots for features (using scatterplotmatrix()). In order to add another dimension to the scatter plots, we can also assign different colors for different target classes.

from mlxtend.plotting import scatterplotmatrix

fig, axes = scatterplotmatrix(X[y==0], figsize=(34, 30), alpha=0.5)
fig, axes = scatterplotmatrix(X[y==1], fig_axes=(fig, axes), alpha=0.5)
fig, axes = scatterplotmatrix(X[y==2], fig_axes=(fig, axes), alpha=0.5, names=attribute_names)

A matrix of scatter plot of all wine attributes with different colors for wine types

By the way, for plotting similar scatter plots, you can also use Pandas’ scatter_matrix() or seaborn’s pairplot() function.

Bootstrapping

The bootstrap is an easy way to estimate a sample statistic and generate the corresponding confidence interval by drawing random samples with replacement. For this, you can use the bootstrap() function from the library. Note that you can pass a custom statistic to the bootstrap function through argument func. The custom function must return a scalar value.

from mlxtend.evaluate import bootstrap

# Generating 100 random data with a mean of 5
random_data = np.random.RandomState(123).normal(loc=5., size=100)

avg, std_err, ci_bounds = bootstrap(
    random_data, 
    num_rounds=1000, 
    func=np.mean,  # A function to compute a sample statistic can be passed here
    ci=0.95, 
    seed=123
)

print(
    f"Mean: {avg.round(2)} \n"
    f"Standard Error: +/- {std_err.round(2)} \n"
    f"CI95: [{ci_bounds[0].round(2)}, {ci_bounds[1].round(2)}]"
)

>>> Mean: 5.03
>>> Standard Error: +/- 0.11
>>> CI95: [4.8, 5.26]

1. Sign up to Heroku & Deploy Your First PostgreSQL Database

You can signup for free to Heroku. After signing up and logging into your account, you will be directed to the Heroku Dashboard. Then, you can follow the instructions in the following clip to create a new app and add a PostgreSQL database.

The free plan allows you to have a maximum of 20,000 rows of data and up to 20 connections to the database. This plan is usually enough for a small personal project.

To address the issue of occasional changes in the database credentials, we can use Heroku CLI to retrieve the database URL dynamically. But first, let’s go over the procedure for logging in to your account via Heroku CLI.

2. Access your Heroku account using Token

What’s covered in this section is applicable in general for working with any Heroku applications through Heroku CLI.

$heroku authorization:create

export HEROKU_API_KEY=<your_token>

$heroku config:get DATABASE_URL --app <your-app-name>

import subprocess
heroku_app_name = "your-app-name"
raw_db_url = subprocess.run(
    ["heroku", "config:get", "DATABASE_URL", "--app", heroku_app_name],
    capture_output=True  # capture_output arg is added in Python 3.7
).stdout 

Create SQLAlchemy Engine

Before we ingest data to a table in the deployed PostgreSQL database using Pandas, we have to create an SQLAlchemy engine that will be passed to the Pandas method. The SQLAlchemy engine/connection can be created using the following code snippet:

import subprocess
from sqlalchemy.engine.create import create_engine

# Get the Database URL using Heroku CLI
# -------------------------------------
# Running the following from Python: $heroku config:get DATABASE_URL --app your-app-name
heroku_app_name = "your-app-name"

# Assumption: HEROKU_API_KEY is set in your terminal
# You can confirm that it's set by running the following python command os.environ["HEROKU_API_KEY"]
raw_db_url = subprocess.run(
    ["heroku", "config:get", "DATABASE_URL", "--app", heroku_app_name],
    capture_output=True  # capture_output arg is added in Python 3.7
).stdout 

# Convert binary string to a regular string & remove the newline character
db_url = raw_db_url.decode("ascii").strip()

# Convert "postgres://<db_address>"  --> "postgresql+psycopg2://<db_address>" needed for SQLAlchemy
final_db_url = "postgresql+psycopg2://" + db_url.lstrip("postgres://")  # lstrip() is more suitable here than replace() function since we only want to replace postgres at the start!

# Create SQLAlchemy engine
# ------------------------
engine = create_engine(final_db_url)

As you may note in the above, some string manipulation is required before creating the SQLAlchemy engine.

Ingest Data using Pandas & SQLAlchemy

We can ingest the data into a table by simply using pandas to_sql() function and passing the SQLAlchemy engine/connection object to it.

import pandas as pd
from sqlalchemy.types import Integer, DateTime

DATA_URL = "https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/latest/owid-covid-latest.csv"
df = pd.read_csv(DATA_URL)

df.to_sql(
    "covid19",  # table name
    con=engine,
    if_exists='replace',
    index=False,  # In order to avoid writing DataFrame index as a column
    dtype={
        "last_updated_date": DateTime(),
        "total_cases": Integer(),
        "new_cases": Integer()
    }
)

In the above example, the data type of few columns is specified. You can determine the dtype of columns by passing a dictionary in which keys should be the column names and the values should be the SQLAlchemy types. For all available dtypes, you can check SQLAlchemy documentation for the data types it supports.

Signup

You can sign up at getdbt.com. The free plan is a great plan for small projects and testing.

Database with populated data

You can check my post on how to deploy a free PostgreSQL database on Heroku. The post provides step-by-step instructions on how to do it.

You can also check the data ingestion script in the GitHub repo accompanying this article.

e-alizadeh/sample_dbt_project

Following the above, we have generated two tables in a PostgreSQL database that we are going to use in this post. There are two tables in the database, namely covid_latest and population_prosperity. You can find the ingestion script on the GitHub repo for this post.

dbt CLI Installation

You can install the dbt command-line interface (CLI) by following the instructions on the following dbt documentation page.

Installation | dbt Docs

How to use dbt?

A dbt project is a directory containing .sql and .yml files. The minimum required files are:

• A project file named dbt_project.yml: This file contains configurations of a dbt project.
• Model(s) (.sql files): A model in dbt is simply a single .sql file containing a single select statement.

Every dbt project needs a dbt_project.yml file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate on your project.

You can find more information about dbt projects here.

dbt Commands

dbt commands start with dbt and can be executed using one of the following ways:

• dbt Cloud (the command section at the bottom of the dbt Cloud dashboard),
• dbt CLI

Some commands can only be used in dbt CLI like dbt init. Some dbt commands we will use in this post are

• dbt init (only in dbt CLI)
• dbt run
• dbt test
• dbt docs generate

Step 1: Initialize a dbt project (sample files) using dbt CLI

You can use [dbt init](https://docs.getdbt.com/reference/commands/init) to generate sample files/folders. In particular, dbt init project_name will create the following:

• a ~/.dbt/profiles.yml file if one does not already exist
• a new folder called [project_name]
• directories and sample files necessary to get started with dbt

The result is a directory with the following sample files.

sample_dbt_project
├── README.md
├── analysis
├── data
├── dbt_project.yml
├── macros
├── models
│   └── example
│       ├── my_first_dbt_model.sql
│       ├── my_second_dbt_model.sql
│       └── schema.yml
├── snapshots
└── tests

For this post, we will just consider the minimum files and remove the extra stuff.

sample_dbt_project
├── README.md
├── dbt_project.yml
└── models
    ├── my_first_dbt_model.sql
    ├── my_second_dbt_model.sql
    └── schema.yml

Step 2: Set Up a Git Repository

You can use an existing repo, as specified during the setup. You can configure the repositories by following the dbt documentation here.

git init
git add .
git commit -m "first commit"
git remote add origing <repo_url>
git push -u origin master

Step 3: Set Up a New Project on dbt Cloud Dashboard

In the previous step, we created a sample dbt project containing sample models and configurations. Now, we want to create a new project and connect our database and repository on the dbt Cloud dashboard.

Before we continue, you should have

• some data already available in a database,
• a repository with the files generated at the previous step

You can follow the steps below to set up a new project in dbt Cloud (keep in mind this step is different than the previous step in that we only generated some sample files).

The dbt_project.yml file for our project is shown below (you can find the complete version in the GitHub repo to this post).

name: 'my_new_project'
version: '1.0.0'
config-version: 2

vars:
  selected_country: USA
    selected_year: 2019

# This setting configures which "profile" dbt uses for this project.
profile: 'default'

# There are other stuff that are generated automatically when you run `dbt init`

dbt models

Let’s create simple dbt models that retrieve few columns of the tables.

select "iso_code", "total_cases", "new_cases" from covid_latest

select "code", "year", "continent", "total_population" from population_prosperity

Jinja & Macros

dbt uses Jinja templating language, making a dbt project an ideal programming environment for SQL. With Jinja, you can do transformations that are not normally possible in SQL, like using environment variables, or macros — abstract snippets of SQL, which is analogous to functions in most programming languages. Whenever you see a { ... }, you’re already using Jinja. For more information about Jinja and additional Jinja-style functions defined, you can check dbt documentation.

Later in this post, we will cover custom macros defined by dbt.

Using Variables

name: 'my_new_project'
version: '1.0.0'
config-version: 2

vars:
  selected_country: USA
    selected_year: 2019

Macros

There are many useful transformations and useful macros in dbt_utils that can be used in your project. For a list of all available macros, you can check their GitHub repo.

Now, let’s add dbt_utils to our project and install it by following the below steps:

1. Add dbt_utils macro to your packages.yml file, as follows:

packages:
  - package: dbt-labs/dbt_utils
    version: 0.6.6

2. Run dbt deps to install the package.

Complex dbt models

The models (selects) are usually stacked on top of one another. For building more complex models, you will have to use [ref()](https://docs.getdbt.com/reference/dbt-jinja-functions/ref) macro. ref() is the most important function in dbt as it allows you to refer to other models. For instance, you may have a model (aka SELECT query) that does multiple stuff, and you don’t want to use it in other models. It will be difficult to build a complex model without using macros introduced earlier.

select *
from {{ref('population')}} 
inner join {{ref('covid19_latest_stats')}} 
on {{ref('population')}}.code = {{ref('covid19_latest_stats')}}.iso_code 
where code='{{ var("selected_country") }}' AND year='{{ var("selected_year") }}'

select
  continent,
  {{ dbt_utils.pivot(
      "population.year",
      dbt_utils.get_column_values(ref('population'), "year")
  ) }}
from {{ ref('population') }}
group by continent

select
  continent,
    sum(case when population.year = '2015' then 1 else 0 end) as "2015",
        sum(case when population.year = '2017' then 1 else 0 end) as "2017",
        sum(case when population.year = '2017' then 1 else 0 end) as "2016",
        sum(case when population.year = '2017' then 1 else 0 end) as "2018",
        sum(case when population.year = '2017' then 1 else 0 end) as "2019"
from "d15em1n30ihttu"."dbt_ealizadeh"."population"
group by continent
limit 500
/* limit added automatically by dbt cloud */

Database administration using Hooks & Operations

There are database management tasks that require running additional SQL queries, such as:

• Create user-defined functions
• Grant privileges on a table
• and many more

dbt has two interfaces (hooks and operations) for executing these tasks and importantly version control them. Hooks and operations are briefly introduced here. For more info, you can check dbt documentation.

1. Project environment

You need to have your project environment managed in Poetry since we will be using the pyproject.toml file to build our package and publish it.

2. Package repository

We will need a package repository to host the Python package; the most popular one is PyPI. So, if you want to publish your library on PyPI, you need to first create an account on PyPI.

Step 1: Build your package

Once you have your Python package ready to be published, you first need to build your package using the following command from the directory that contains the pyproject.toml file:

poetry build

The above command will create two files in the dist (distribution) directory. A folder will be created if there is no dist folder.

First, a source distribution (often known as sdist) is created that is an archive of your package based on the current platform (.tar.gz for Unix and .zip for Windows systems)¹.

In addition to sdist, poetry build creates a Python wheel (.whl) file. In a nutshell, a Python wheel is a ready-to-install format allowing you to skip the build stage, unlike the source distribution. A wheel filename is usually in the following format²:

{pkg-name}-{pkg-version}(-{build}?)-{python-implementation}-{application binary interface}-{platform}.whl

From the above figure, the package I built is called pypocket with version 0.2.0 in Python 3 that is not OS-specific (none ABI) and suitable to run on any processor architecture.

Step 2: Publish your package

Once the package is built, you can publish it on PyPI (or other package repositories).

poetry config repositories.testpypi https://test.pypi.org/legacy/

poetry publish -r testpypi

poetry publish

Related posts

A Guide to Python Environment, Dependency and Package Management: Conda + Poetry - Personal Website & Blog

A Quick Note on Package Repositories

The most popular Python package repository is the Python Package Index (PyPI), a public repository for many Python libraries. You can install packages from PyPI by running pip install package_name. Python libraries can also be packaged using conda, and a popular host for conda packages is Anaconda. You can install conda packages by running conda install package_name in your conda environment.

Issues with conda

I think conda tries to do too much. After several years of using conda, here are few of my observations on conda as a package and dependency management:

pyproject.toml: Python Configuration file

pyproject.toml file is a new Python configuration file defined in PEP518 to store build system requirements, dependencies, and many other configurations. You can even replace setup.cfg and setup.py files in most scenarios. You can save most configurations related to specific python packages like pytest, coverage, bumpversion, Black code styling, and many more in a single pyproject.toml file. You previously had to either write those configurations in individual files or other configuration files like setup.cfg. However, pyproject.toml can include all of them and also all project package requirements too.

Step 1: Create a minimal conda environment

You can create a conda environment from the following YAML file by running conda env create -f environment.yaml. This will create a fresh conda environment that has Python 3.8. In a conda environment, you can pass a list of channels (the order is important) from which you want to install your packages. In addition to the default channel on Anaconda Cloud that is curated by Anaconda Inc., there are other channels that you can install packages. A popular channel is conda-forge that includes a community-led collection of packages. If you have a private conda channel, you can write it in the channels section.

name: post

channels:
  - default
  - conda-forge

dependencies:
  - python=3.8

Step 2: Install Poetry tool

You can install Poetry as per their instruction here. The recommended way is to install Poetry using the following command for OSx, Linux, or WSL (Windows Subsystem Linux).

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

Note: Installing Poetry using the preferred approach that is by the custom installer (the first approach that downloads get-poetry.py script) will install Poetry isolated from the rest of the system.

⚠️ Although not recommended, there is also a pip version of Poetry that you can install (pip install poetry). The developers warn against using the pip version in the documentation since it might cause some conflicts with other packages in the environment. But, if our environment is basically empty (although some base packages are installed like pip when creating a conda environment), then it is probably fine to install it through pip!

Step 3: Configure your Poetry

To configure Poetry for a new project, Poetry makes it very easy to create a configuration file with all your desired settings. You can interactively create a pyproject.toml file by simply running poetry init. This will prompt few questions about the desired Python packages you want to install. You can press Enter to process with default options.

As you can see in the above screenshot, you can add some packages only for development dependencies. Initializing the Poetry for your project will create the pyproject.toml file that includes all configurations we defined during the setup. We have one main section for all dependencies (used in both production and development environments), but we also have a section that contains packages used mainly for development purposes like pytest, sphinx, etc. This is the other advantage over other dependency management tools. You only need one configuration file for both your production and development environments.

[tool.poetry]
name = "my_package"
version = "0.0.1"
description = ""
authors = ["ealizadeh <abc@edf.com>"]

[tool.poetry.dependencies]
python = "^3.8"
requests = "^2.25.1"

[tool.poetry.dev-dependencies]
pytest = "^6.2.1"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.pytest.ini_options]
minversion = "6.0"
addopts = "-ra -q"
testpaths = [
    "tests",  # You should have a "tests" directory
]

Step 4: Installing dependencies

Once you have your dependencies and other configurations in a pyproject.toml file, you can install the dependencies by simply running

poetry install

This will create a poetry.lock file. This file basically contains the exact versions of all the packages locking the project with those specific versions. You need to commit both the pyproject.toml file and poetry.lock file. I would strongly recommend you not to update the poetry.lock file manually. Let poetry does its magic!!

Add new packages

If you want to add (or remove) a package to your environment, I would highly recommend you to do so by using the following command:

poetry add package_name

This will automatically add the package name and version to your pyproject.toml file and updates the poetry.lock accordingly. poetry add takes care of all dependencies, and adds the package in the [tool.poetry.dependencies] section.

If you want to add a package to your development environment, you can simly pass a --dev option as below:

poetry add package_name --dev

You can specify a specific version of a package, or even adding a package through git+https or git+ssh (see here for more details).

Remove packages

You can remove a package as following:

poetry remove package_to_remove

Show package dependencies

If you want to see a list of all installed packages in your environment, you can run the following command:

poetry show

Note that this will show the package dependencies too. It is sometimes helpful to see the dependencies of a Python package. Fortunately, you can do so using poetry show. For instance, we can see the list of dependencies of requests package in our environment using the following command:

poetry show requests

Even better, you can see all your project’s dependencies by just running:

poetry show --tree

From above figure, you can see that the blue-font package names (requests and pytest) are explicitly added to pyproject.toml file. Other libraries, in yellow, are their dependencies and do not need to be in your toml file.

Mean

The arithmetic mean, or simply mean or average is probably the most popular estimate of location. There different variants of mean, such as weighted mean or trimmed/truncated mean. You can see how they can be computed below.

denotes the total number of observations (rows).

Weighted mean (equation 1.2) is a variant of mean that can be used in situations where the sample data does not represent different groups in a dataset. By assigning a larger weight to groups that are under-represented, the computed weighted mean will more accurately represent all groups in our dataset.

Another variant of mean is the trimmed mean (eq. 1.3) that is a robust estimate. This metric is used in calculating the final score in many sports where a panel of judges will each give a score. Then the lowest and the highest scores are dropped and the mean of the remaining scores are computed as a part of the final score¹. One such example is in the international diving score system.

Median

Median is the middle of a sorted list, and it’s a robust estimate. For an ordered sequence , the median is computed as follows:

Analogous to the weighted mean, we can also have the weighted median that can be computed as follows for an ordered sequence with weights where .

Mode

The mode is the value that appears most often in the data and is typically used for categorical data, and less for numeric data see [1].

Python Implementation

Let’s first import all necessary Python libraries and generate our dataset.

import pandas as pd
import numpy as np
from scipy import stats
import robustats

df = pd.DataFrame({
    "data": [2, 1, 2, 3, 2, 2, 3, 20],
    "weights": [1, 0.5, 1, 1, 1, 1, 1, 0.5] # Not necessarily add up to 1!!
})
data, weights = df["data"], df["weights"]

You can use NumPy’s average() function to calculate the mean and weighted mean (equations 1.1 & 1.2). For computing truncated mean, you can use trim_mean() from the SciPy stats module. A common choice for truncating the top and bottom of the data is 10%see [1].

You can use NumPy’s [median()](https://numpy.org/doc/stable/reference/generated/numpy.median.html) function to calculate the median. For computing the weighted median, you can use weighted_median() from the robustats Python library (you can install it using pip install robustats)². Robustats is a high-performance Python library to compute robust statistical estimators implemented in C.

For computing the mode, you can either use the mode() function either from the robustats library that is particularly useful on large datasets or from scipy.stats module.

mean = np.average(data) # You can use Pandas dataframe.mean()
weighted_mean = np.average(data, weights=weights)
truncated_mean = stats.trim_mean(data, proportiontocut=0.1)
median = np.median(data) # You can use Pandas dataframe.median()
weighted_median = robustats.weighted_median(x=data, weights=weights)
mode = stats.mode(data)  # You can also use robustats.mode() on larger datasets

print("Mean: ", mean.round(3))
print("Weighted Mean: ", weighted_mean.round(3))
print("Truncated Mean: ", truncated_mean.round(3))
print("Median: ", median)
print("Weighted Median: ", weighted_median)
print("Mode: ", mode)

>>> Mean:  4.375
>>> Weighted Mean:  3.5
>>> Truncated Mean:  4.375
>>> Median:  2.0
>>> Weighted Median:  2.0
>>> Mode:  ModeResult(mode=array([2]), count=array([4]))

Now, let’s see if we just remove 20 from our data, how that will impact our mean.

mean = np.average(data[:-1]) # Remove the last data point (20)
print("Mean: ", mean.round(3))

>>> Mean:  2.143

You can see how the last data point (20) impacted the mean (4.375 vs 2.143). There can be many situations that we may end up with some outliers that should be cleaned from our datasets like faulty measurements that are in orders of magnitude away from other data points.

Mean Absolute Deviation

One way to get this estimate is to calculate the difference between the largest and the lowest value to get the range. However, the range is, by definition, very sensitive to the two extreme values. Another option is the mean absolute deviation that is the average of the sum of all absolute deviation from the mean, as can be seen in the below formula:

One reason why the mean absolute deviation receives less attention is since mathematically it’s preferable not to work with absolute values if there are other desirable options such as squared values available for instance, is differentiable everywhere while the derivative of   is not defined at .

Variance & Standard Deviation

The variance and standard deviation are much more popular statistics than the mean absolute deviation to estimate the data dispersion.

As can be noted from the formula, the standard deviation is on the same scale as the original data making it an easier metric to interpret than the variance. Analogous to the trimmed mean, we can also compute the trimmed/truncated standard deviation that is less sensitive to outliers.

A good way of remembering some of the above estimates of variability is to link them to other metrics or distances that share a similar formulation see [1]. For instance,

Median Absolute Deviation (MAD)

Like the arithmetic mean, none of the estimates of variability (variance, standard deviation, mean absolute deviation) is robust to outliers. Instead, we can use the median absolute deviation from the median to check how our data is spread out in the presence of outliers. The median absolute deviation is a robust estimator, just like the median.

Percentiles

Percentiles (or quantiles) is another measure of the data dispersion that is based on order statistics (statistics based on sorted data). -th percentile is the least percentage of the values that are lower than or equal to percent.

25th (Q1) and 75th (Q3) percentiles are particularly interesting since their difference (Q3 – Q1) shows the middle 50% of the data. The difference is known as the interquartile range (IQR) (IQR=Q3-Q1). Percentiles are used to visualize data distribution using boxplots.

A nice article about boxplots is available on Towards Data Science blog.

Python Implementation

You can use NumPy’s var() and std() function to calculate the variance and standard deviation, respectively. On the other hand, to calculate the mean absolute deviation, you can use Pandas mad() function. For computing the trimmed standard deviation, you can use SciPy’s tstd() from the stats module. You can use Pandas boxplot() to quickly visualize a boxplot of the data.

import pandas as pd
import numpy as np
from scipy import stats

variance = np.var(data)
standard_deviation = np.std(data)  # df["Population"].std()
mean_absolute_deviation = df["data"].mad()
trimmed_standard_deviation = stats.tstd(data)
median_absolute_deviation = stats.median_abs_deviation(data, scale="normal")  # stats.median_absolute_deviation() is deprecated

# Percentile
Q1 = np.quantile(data, q=0.25)  # Can also use dataframe.quantile(0.25)
Q3 = np.quantile(data, q=0.75)  # Can also use dataframe.quantile(0.75)
IQR = Q3 - Q1

print("Variance: ", variance.round(3))
print("Standard Deviation: ", standard_deviation.round(3))
print("Mean Absolute Deviation: ", mean_absolute_deviation.round(3))
print("Trimmed Standard Deviation: ", trimmed_standard_deviation.round(3))
print("Median Absolute Deviation: ", median_absolute_deviation.round(3))
print("Interquantile Range (IQR): ", IQR)

>>> Variance:  35.234
>>> Standard Deviation:  5.936
>>> Mean Absolute Deviation:  3.906
>>> Trimmed Standard Deviation:  6.346
>>> Median Absolute Deviation:  0.741
>>> Interquantile Range (IQR):  1.0

NeuralProphet vs. Prophet

From the library name, you may ask what is the main difference between Facebook’s Prophet library and NeuralProphet. According to NeuralProphet’s documentation, the added features are:

• Using PyTorch’s Gradient Descent optimization engine making the modeling process much faster than Prophet
• Using AR-Net for modeling time-series autocorrelation (aka serial correlation)
• Custom losses and metrics
• Having configurable non-linear layers of feed-forward neural networks,
• etc.

Project Maintainers

Based on the project’s GitHub page, the main maintainer of this project is Oskar Triebe from Stanford University with collaboration from Facebook and Monash University.

Installation

The project is in the beta phase, so I would advise you to be cautious if you want to use this library in a production environment.

You can install the package using pip install neuralprophet. However, if you are going to use the package in a Jupyter Notebook environment, you should install their live version pip install neuralprophet[live]. This will provide more features such as a live plot of train and validation loss using plot_live_loss().

git clone https://github.com/ourownstory/neural_prophet
cd neural_prophet
pip install .[live]

I would recommend creating a fresh environment (a conda or venv) and installing the NeuralProphet package from the new environment letting the installer take care of all dependencies (it has Pandas, Jupyter Notebook, PyTorch as dependencies).

Now that we have the package installed, let’s play!

Implementation with a Case Study

Here, I’m using the daily climate data in Delhi from 2013 to 2017 that I found on Kaggle. First, let’s import the main packages.

import pandas as pd
from neuralprophet import NeuralProphet

Then, we can read the data into a Panda DataFrame. NeuralProphet object expects the time-series data to have a date column named ds and the time-series column value we want to predict as y.

# Data is from https://www.kaggle.com/sumanthvrao/daily-climate-time-series-data
df = pd.read_csv("./DailyDelhiClimateTrain.csv", parse_dates=["date"])
df = df[["date", "meantemp"]]
df.rename(columns={"date": "ds", "meantemp": "y"}, inplace=True)

Now let’s initialize the model. Below, I’ve brought all default arguments defined for the NeuralProphet object, including additional information about some. These are the hyperparameters you can configure in the model. Of course, if you are planning to use the default variables, you can just do model = NeuralProphet().

# model = NeuralProphet() if you're using default variables below.
model = NeuralProphet(
    growth="linear",  # Determine trend types: 'linear', 'discontinuous', 'off'
    changepoints=None, # list of dates that may include change points (None -> automatic )
    n_changepoints=5,
    changepoints_range=0.8,
    trend_reg=0,
    trend_reg_threshold=False,
    yearly_seasonality="auto",
    weekly_seasonality="auto",
    daily_seasonality="auto",
    seasonality_mode="additive",
    seasonality_reg=0,
    n_forecasts=1,
    n_lags=0,
    num_hidden_layers=0,
    d_hidden=None,     # Dimension of hidden layers of AR-Net
    ar_sparsity=None,  # Sparcity in the AR coefficients
    learning_rate=None,
    epochs=40,
    loss_func="Huber",
    normalize="auto",  # Type of normalization ('minmax', 'standardize', 'soft', 'off')
    impute_missing=True,
    log_level=None, # Determines the logging level of the logger object
)

After configuring the model and its hyperparameters, we need to train the model and make predictions. Let’s make up to a one-year prediction of the temperature.

metrics = model.fit(df, validate_each_epoch=True, freq="D")
future = model.make_future_dataframe(df, periods=365, n_historic_predictions=len(df))
forecast = model.predict(future)

You can simply plot the forecast by calling model.plot(forecast) as following:

fig, ax = plt.subplots(figsize=(14, 10))
model.plot(forecast, xlabel="Date", ylabel="Temp", ax=ax)
ax.set_title("Mean Temperature in Delhi", fontsize=28, fontweight="bold")

The one-year forecast plot is shown below, where the time period between 2017-01-01 to 2018-01-01 is the prediction. As can be seen, the forecast plot resembles the historical time-series. It both captured the seasonality as well as the slow-growing linear trend.

You can plot the parameters by calling model.plot_parameters()

The model loss using Mean Absolute Error (MAE) is plotted below. You can also use the Smoothed L1-Loss function.

fig, ax = plt.subplots(figsize=(14, 10))
ax.plot(metrics["MAE"], 'ob', linewidth=6, label="Training Loss")  
ax.plot(metrics["MAE_val"], '-r', linewidth=2, label="Validation Loss")

# You can use metrics["SmoothL1Loss"] and metrics["SmoothL1Loss_val"] too.

1. Base-Rate Neglect

This fallacy is a common reasoning error where people neglect the distribution of the data in favor of specific individual information. Here is an example of this bias from the book.

Mark is a man from Germany who wears glasses and listens to Mozart. Which one is more likely?

He is 1) a truck driver or 2) a literature professor in Frankfurt?

Most people will bet on Option 2 (the wrong option). The number of truck drivers in Germany is 10,000 times more than the number of literature professors in Frankfurt. Hence, it’s more likely that Mark is a truck driver see [1].

2. Clustering Illusion

The brain seeks patterns.

The brain seeks patterns, coherence, and order where none really exists. As per this cognitive bias, we are oversensitive to finding a structure or rule. The human’s false pattern recognition is also known as apophenia. That is the tendency to make meaningful connections between unrelated things. Examples of such phenomena are gambler’s fallacy, figures in clouds, and patterns with no deliberate designs. A popular example of this is the “Face on Mars” as shown below:

This is prominent in the gambling, misinterpretation of statistics, conspiracy theories, etc. This cognitive error is also linked to two other errors: coincidence and false causality. The way to overcome the clustering illusion, particularly for anyone working with data, is to assume any pattern found in the data as random and statistically test the pattern found see [2].

3. Confirmation Bias

The confirmation bias is the only cognitive error that the book author wrote two parts about it. The following quote from the book says it all:

The confirmation bias is the mother of all misconceptions. —Rolf Dobelli

Confirmation bias is the tendency to interpret new information based on our existing beliefs and discard any opposing evidence (disconfirming evidence). The internet is the main ground for this bias. Think about the YouTube video suggestions made to us based on our watch history. It gives us recommendations to videos that align with our current views/beliefs/theories. We are actually prone to confirmation bias in many platforms that use Recommender systems like Google, Facebook, Twitter, etc.

4. Overconfidence Effect

This cognitive bias tells us that we systematically overestimate our ability to predict. In other words, this bias says that we believe subjectively that our judgment is better than it objectively is see [1]. No wonder why most major projects are not completed in less time or cheaper than initially predicted. The overconfidence bias is more common among experts. The reason is that experts obviously know more about their own field, but they overestimate exactly how much more.

5. Regression to Mean

Imagine the weather in your city reaches a record hot weather. The temperature will most likely drop in the next few days, back towards the monthly average. This bias depends on the random variance impacting any measurement, causing some to be extreme. Ignoring this bias leads to overestimating the correlation between the two measures. For instance, if an athlete performs extremely well in a year, we expect a better performance the year after. If that’s not the case, we may come up with causal relationships instead of considering that we probably overestimated the next year’s performance!

6. Induction

The inductive bias occurs when we draw universal conclusions from individual observations. For example, all observed people in a city wear glasses. Therefore all the people in that city wear glasses. This can lead to false causality (another bias I will talk about later). For example, every time Bob drinks milk, he gets cramps, and therefore he concludes that he gets cramps because he drinks milk!

7. Intention-To-Treat Error

Mainly used in medical research studies, the intention-to-treat (ITT) principle is crucial in interpreting the results of randomized clinical trials. It helps the researchers to assess the true effect of choosing a medical treatment. Let’s have an example to understand this fallacy better.

Suppose a pharmaceutical company developed a new drug for heart diseases. A study “proves” that the drug is improving the patient’s health and reduces the chance of dying from heart diseases. The five-year mortality rate of patients who take the drug regularly is 15%. The company may not tell you that the mortality rate of patients who took the drug irregularly was 30%. So, is the drug a complete success?

The point here is that the drug may not be the decisive factor here, but the patient’s behavior. The patients who didn’t take the drug according to the schedules may have side effects causing them to stop taking the drug. Hence, these patients will not be in the “regular category” of the study, for which a 15% rate was reported. So, because of the ITT error, the drug looks much more effective than it actually is see [1].

8. False Causality

The Correlation does not imply causation.

This fallacy occurs when we wrongly infer a cause-and-effect relationship between two measurements solely based on their correlation. For instance, the per capita consumption of mozzarella cheese is highly correlated with civil engineering PhDs awarded (r=0.9586). Does this mean that the consumption of mozzarella cheese leads to more civil engineering doctorates awarded?

9. The Problem with Averages

Don’t cross a river if it is (on average) four feet deep. —Nassim Taleb

Working with averages may mask the underlying distribution. An outlier may significantly change the picture. This is important for anyone that works with data. A few extreme outliers may dominate.

10. Information Bias

This is when additional information does not add any value to the action or decision you are taking. Extra information does not guarantee better decisions, and at times it may actually put you at a disadvantage in addition to wasting time and money. Observational studies may be more prone to this type of bias, mainly because they rely on self-reporting data collection, although introducing randomness in interventional studies may reduce that. Missing data can be another reason for introducing information bias².

11. The Law of Small Numbers

This law refers to incorrect reasoning that a small sample drawn from a population resembles the overall population. This is particularly important in statistical analysis. In his book “Thinking, Fast and Slow” see [3], the Nobel prize winner Daniel Kahneman points out that even professionals and experts sometimes fall into this fallacy.

Suppose you have a bag of 1000 marbles, of which 500 are red, and the other 500 marbles are black. Without looking, you draw three marbles, and all turn out to be black. You may infer that all marbles in the bag are black. In this scenario, you’ve fallen into the fallacy of the law of small numbers.

The best way to counter this fallacy, as you may have guessed by now, is to use the Law of Large Numbers to have a sample size that resembles the overall population.

12. Ambiguity Aversion (Risk vs Uncertainty)

Suppose we have two bags. Bag A has 50 black and another 50 red marbles. Bag B also has 100 marbles (red and black), but we don’t know how many are for each color. Now, if you pull a red marble from a bag (of course without taking a look!!), you will win $100. So, do you choose bag A or bag B?

The majority will go with bag A. Now, let’s repeat the experiment, and this time if you pull a black marble from a bag, you will win $100. In this case, the majority will pick bag A too. This illogical result is known as Ellsberg Paradox, stating that people prefer known probabilities over unknown (ambiguous) ones see [1]. We should be aware of the difference between risks and uncertainties. These two terms are used interchangeably. However, there is a significant difference between the two.

Although difficult, tolerating ambiguity/uncertainty may help us to handle this fallacy better.

13. Planning Fallacy

Have you wondered why you always underestimate the amount of time it takes you to finish a task or a project? You may fall into the planning fallacy. This cognitive error says that our planning is usually very ambitious. The two main reasons behind this fallacy are 1) wishful thinking and 2) neglecting the external influences. One way we can better plan is to have a premortem session that you can go over similar projects or consider outside influences beforehand.

14. Déformation Professionnelle

If your only tool is a hammer, all your problems will be nails. —Mark Twain

This fallacy is in place when you view things from one’s own profession instead of a broader perspective. This may impact anyone who learns something new and uses that in every situation. Let’s say you just learned about a new machine learning model. You may use it in cases that better models are available. I actually did the same. During my studies, I learned about support vector machine (SVM) (a binary classifier). Whenever I had a classification problem, I would initially think of using SVM, even in multiclass classification for which SVM may not be the best option to start for.

15. Exponential Growth Bias

Unlike linear growths, exponential growths are not intuitive to understand. Consider the following two options:

1. You will get $1,000 every day for the next 30 days.
2. You will get a cent on the first day, two cents on the second day, and the amount gets doubled every day for the next 30 days.

Which one do you choose? Option 1 will earn you $30,000 at the end, whereas Option 2 will get you over $10.7 million (you can calculate it yourself using where (1 cent), (doubling), ).

We can use a logarithmic scale to transform the original data for a better understanding.

Time-Series Data Modeling using PAR

A probabilistic autoregressive (PAR) model is used to model multi-type multivariate time-series data. The SDV library has this model implemented in the PAR class (from time-series module).

Let’s work out an example to explain different arguments of PAR class. We are going to work with a time-series of temperatures in multiple cities. The dataset will have the following column: Date, City, Measuring Device, Where, Noise.

In PAR, there are four types of columns considered in a dataset.

1. Sequence Index: This is the data column with the row dependencies (should be sorted like datetime or numeric values). In time-series, this is usually the time axis. In our example, the sequence index will be the Date column.

2. Entity Columns: These columns are the abstract entities that form the group of measurements, where each group is a time-series (hence the rows within each group should be sorted). However, rows of different entities are independent of each other. In our example, the entity column(s) will be only the City column. By the way, we can have more columns as the argument type should be a list.

3. Context Columns: These columns provide information about the time-series’ entities and will not change over time. In other words, the context columns should be constant within groups. In our example, Measuring Device and Where are the context columns.

4. Data Columns: Any other columns that do not belong to the above categories will be considered data columns. The PAR class does not have an argument for assigning data columns. So, the remaining columns that are not listed in any of the previous three categories will automatically be considered data columns. In our example, the Noise column is the data column.

import pandas as pd
from sdv.timeseries import PAR

actual_data = pd.read_csv("./daily_min_temperature_data_melbourne.csv")
actual_data["Date"] = pd.to_datetime(actual_data["Date"])

# Define -> Fit -> Save a PAR model
model = PAR(sequence_index="Date")
model.fit(actual_data)
model.save("data_generation_model_single_city.pkl")

# After a model is trained and pickled.
We can comment above code and just load the model next time.

model = PAR.load("data_generation_model_single_city.pkl")

# Generate new data
new_data = model1.sample(num_sequences=1)   # may take few seconds to generate
new_data["Date"] = pd.to_datetime(new_data["Date"])

# Compare descriptive statistics
stat_info1 = actual_data.describe().rename(columns={"Temp": "Real Data"})
stat_info2 = new_data.describe().rename(columns={"Temp": "PAR Generated Data"})

print(stat_info1.join(stat_info2))

stat_info1.join(stat_info2)

import pandas as pd
from sdv.timeseries import PAR

all_data = pd.read_csv("./fake_time_series_data_multiple_cities.csv")
all_data["Date"] = pd.to_datetime(all_data["Date"])

# Define -> Fit -> Save a PAR model
model = PAR(
    entity_columns=["City"],
    context_columns=["Measuring Device", "Location"],
    sequence_index="Date",
)
model.fit(all_data)
model.save("data_generation_model_multiple_city.pkl")

# After a model is trained and pickled. We can comment above code and just load the model next time.

model = PAR.load("data_generation_model_multiple_city.pkl")

# Generate new data for two fake cities
new_data = model1.sample(num_sequences=2)   # may take few seconds to generate

# Compare descriptive statistics
stat_info1 = all_data.describe().rename(columns={"Temp": "Original Data"})
stat_info2 = new_cities[new_cities["City"] == cities[0]].describe().rename(columns={"Temp": f"PAR Generated Data {cities[0]}"})
stat_info3 = new_cities[new_cities["City"] == cities[1]].describe().rename(columns={"Temp": f"PAR Generated Data {cities[1]}"})
stat_info4 = all_data[all_data["City"] == "City A"].describe().rename(columns={"Temp": "Original Data (City A)"})
stat_info5 = all_data[all_data["City"] == "City B"].describe().rename(columns={"Temp": "Original Data (City B)"})
stat_info6 = all_data[all_data["City"] == "City C"].describe().rename(columns={"Temp": "Original Data (City C)"})

stat_comparison = stat_info1.join(stat_info2).join(stat_info3).join(stat_info4).join(stat_info5).join(stat_info6)
stat_comparison