Ritik Kumar

I Analyzed 300+ Indian IPOs. Here’s What Actually Works.

2025-12-31T00:00:00+00:00

In India, 103 major IPOs hit the market in 2025 alone. That’s roughly two IPOs every single week. And this happened in a year when the broader markets weren’t exactly on fire. Still, we saw IPOs clock listing gains north of 65% - which is wild, by any standard.

Disclaimer: This post is purely a data-driven analysis. It’s not investment advice. IPO investing carries risk, and past patterns don’t guarantee future gains. Always do your own research and be prepared for potential losses.

For better or worse, IPO investing in India has historically been a mix of data, instinct, and straight-up luck. If you understand how the game works, subscription trends, GMP chatter, and market mood, it’s generally been a decent way to make quick, short-term returns. The playbook is familiar: apply, hope you get an allotment, and exit on listing day to book the pop.

There’s no shortage of opinions and anecdotal wisdom floating around. And if you think heuristically, it’s easy to form thumb rules about how to position yourself. But beyond that, it often comes down to luck and timing.

This post is an attempt to separate intuition from evidence and let the data do the talking.

We’ll dig into questions like:

What actually drives listing gains, and which signals are genuinely useful when betting on IPOs?
When does it make sense to sell on listing day, and when is holding the smarter move?
And finally, how much of the IPO “hype” survives beyond the first trading session?

The Data

The IPO-level data used in this analysis is sourced from Chittorgarh.com, while post-listing, ticker-level price data comes from Fyers. Together, these cover both the IPO phase and what happens after the stock actually starts trading.

I have collected the data of 300+ listings in the Indian market since 2021 for this analysis.

What is GMP?

GMP (Grey Market Premium) is the unofficial price at which IPO shares trade in the grey market before listing. In simple terms, it reflects market excitement around an IPO.

A higher GMP typically indicates stronger demand and expectations of a positive listing, while a low or negative GMP suggests weak interest. It’s not regulated, not guaranteed - but it’s often treated as the pulse of IPO sentiment.

One can find the GMP of a particular stock through websites like Chittorgarh.com

What is Subscription?

Subscription data shows how many times an IPO has been applied for compared to the shares available in that category. This is broken down by investor categories:

QIB - Qualified Institutional Buyers
NII - Non-Institutional Investors (HNIs)
Retail - Individual investors
Employees - If applicable

Oversubscription indicates strong demand, but who is oversubscribing matters just as much as how much.

What Is “IPO Success”, Really?

To keep things objective, this analysis uses a straightforward definition:

Listing success = 1% + return on the listing day

Listing return = (open price - issue price) / issue price

Holding Returns

Performance of the stock after listing, tracked over periods like one week, one month, and one year.

Importantly, predicting direction is much easier than predicting magnitude.

What is Sale Type?

Sale type describes how the company is raising money through the IPO:

Fresh Issue / Fresh Capital → Company is raising new funds to grow or repay debt.
OFS (Offer for Sale) → Existing shareholders (promoters, early investors) are selling their shares.
Combination (Fresh + OFS) → Both new shares are issued, and some existing shareholders are selling.

Why it matters:

Fresh issues signal the company is raising money for growth, while OFS-heavy IPOs are more about exiting investors. This can influence demand, pricing, and post-listing performance.

Correlation

To study how IPO signals relate to listing day returns, two types of correlation are used.

Pearson correlation checks whether two values move together in a straight line. If higher GMP consistently leads to proportionally higher listing gains, Pearson picks that up.

Spearman correlation is more forgiving and more realistic. It only looks at ranking - whether IPOs with higher GMP generally perform better than those with lower GMP.

Why this matters:

A GMP of 40% doesn’t guarantee twice the gains of a GMP at 20%. But IPOs with 40% GMP still tend to outperform IPOs at 20%. Pearson struggles with this nuance; Spearman captures it well.

How to Read Correlation Numbers

Both Pearson r and Spearman ρ range from –1 to +1.

+1 → strong positive relationship
0 → no real relationship
–1 → strong negative relationship


As a rule of thumb:
0.2–0.4 → weak
0.4–0.6 → moderate
0.6–0.8 → strong
0.8+ → very strong

Using both helps distinguish between exact prediction and directional edge, which is far more useful in messy, real-world markets.

Below is the correlation of

Signal	Pearson r	Spearman ρ	What it Tells Us
GMP (%)	0.82	0.76	Strongest predictor by a wide margin
Total Subscription	0.70	0.72	Demand matters, but less than GMP
QIB Subscription	0.67	0.73	Institutional interest is a key signal
NII Subscription	0.58	0.62	HNI demand has moderate impact
Retail Subscription	0.46	0.56	Weakest, but not irrelevant

Clearly, GMP has the best correlation.

To demonstrate this better, below is a plot of GMP, total subscription and listing gains. Note that the x and y axes are log scales (for clarity) and the listing gains are represented by colour (this is rounded to 0.5% to 20% for the sake of clarity)

Clearly, there’s a good chance of listing gains if GMP is 20% or more and/or subscription is 25x or more.

The following section gives a closer look at the same stats.

Listing gains by GMP

Below is the bucketized GMP and listing gains stats. A successful listing is when the listing of an IPO gives more than 1% returns.

gmp_bucket	count	success_rate	median_return
<0	40	0.38	0.00
0–10	99	0.49	0.00
10-15	29	0.62	0.04
15–20	20	0.85	0.14
20–25	21	0.81	0.19
25–30	22	0.82	0.23
30–60	57	0.98	0.37
60+	34	1.00	0.74

Clearly, there’s a good chance of listing gains for cases when GMP is more than 15%

Listing Gains by total subscription

total_sub_bucket	count	success_rate	median_return
<1x	5	0.20	-0.01
1–10x	100	0.43	0.00
10–20x	32	0.62	0.08
20–30x	24	0.50	0.01
30-50x	32	0.81	0.12
50x+	129	0.95	0.36

Here, above 30x total subscription, we stand a decent chance of making good returns

Listing Gains by sale type

sale_type	count	success_rate	median_return
Fresh Capital	57	0.74	0.12
Fresh Capital & OFS	204	0.69	0.10
Offer For Sale	61	0.69	0.08

There’s a slight statistical bias towards fresh capital, which is kinda intuitive. Money going to the company can later be used to pay off debt or for Capex and expansion, etc which is better for the new shareholders

Post listing 1st day return

sale_type	mean	median	count
Fresh Capital	3.77	5.00	57
Fresh Capital & OFS	0.16	-0.14	204
Offer For Sale	-0.11	-0.54	61

Here again, there’s a good bias to fresh capital, meaning it may be worth holding IPOs with fresh capital.

Post listing

The goal of the following table was to understand, statistically, if there is a correlation between listing gains and returns in the 1st week, month and year of listing

listing_gain_bucket	next_week_return mean	next_week_return median	next_week_return count	next_month_return mean	next_month_return median	next_month_return count	next_year_return mean	next_year_return median	next_year_return count
Loss	1.60	-0.26	91	2.02	-1.44	87	8.71	-4.42	58
0–5%	0.79	-1.70	42	3.83	-1.64	42	20.25	1.54	27
5–10%	0.92	-2.03	30	1.11	-1.51	29	6.81	-2.01	21
10–20%	3.95	1.69	49	4.27	-1.06	48	27.87	-0.52	31
20–50%	0.41	-2.90	65	-0.16	-3.23	62	20.51	3.01	48
50%+	-0.07	-2.14	45	-2.52	-5.28	45	11.23	-15.36	41

But clearly, it’s not a good signal. The hold/ buy call based on just listing gains doesn’t make sense.

Average subscription

The data below shows the average ratio between retail and sNII subscriptions. This is a fairly consistent trend: retail investors usually have better chances of getting an allocation based on subscription numbers. However, in terms of absolute returns, investing around Rs2 lakh in the sNII category often offers more upside. On average, the multiple tends to be around 2.5x.

gmp_bucket	sub_sNII (bids below Rs10L)	sub_Retail	sNII_to_Retail_ratio
<0	1.97	3.17	0.62
0–5	4.54	3.13	1.45
5–10	29.21	9.63	3.03
10–20	55.03	21.89	2.51
20–50	106.96	36.40	2.94
50+	150.36	59.76	2.52

Conclusion: Data Over Hype

The numbers reveal a clear pattern: IPO success in India isn’t just a roll of the dice; it’s a signal-driven game. To navigate this market effectively, keep these three data-backed rules in mind:

Trust the GMP: With a 0.76 correlation, the Grey Market Premium is your most reliable North Star. A GMP above 20% is the historical sweet spot for high-probability listing gains.
The 30x Threshold: Subscription matters, but only at scale. Once total demand crosses 30x, the likelihood of a successful listing (1%+ gain) becomes significantly more stable.
Quality of Capital: IPOs focused on Fresh Capital consistently outperform OFS-heavy exits. If the money stays in the company, the market usually rewards it.

The Final Word: A massive listing “pop” does not guarantee long-term growth. The data shows no strong link between Day 1 gains and Year 1 returns. For the short-term trader, follow the signals; for the long-term investor, look beyond the first day’s excitement.

If you have ideas for other comparisons or signals to analyse, I’d love to hear them! Drop your suggestions in the comments, and let’s dig deeper together.

From Announcement to Action: The Impact of Capex News on Stock Prices

2024-11-21T00:00:00+00:00

In the stock market, reacting to news is a well-known tactic. Investors often trade or make swing and long-term positions based on announcements, aiming to capitalize on sharp price movements. But how impactful can a single piece of news be?

This case study focuses on a striking example of how a capital expenditure (capex) announcement transformed market sentiment within hours.

The Company in Focus: Deepak Nitrite

Deepak Nitrite is one of India’s fastest-growing and most trusted names in chemical intermediates. Known for its consistent performance and strategic expansions, the company has built a solid reputation in the industry and the stock market alike.

The Big Day: May 24, 2024

On this day, Deepak Nitrite issued a notification that sent ripples through the market. Investors responded with overwhelming enthusiasm, pushing the stock price up by an impressive 10% by the day’s close.

What Did the Notification Say?

Here’s a snippet from the announcement: [src]

Sub: Intimation under Regulation 30 of SEBI (Listing Obligations and Disclosure Requirements) Regulations, 2015

Pursuant to the requirement of Regutation 30 of SEBI (Listing Obtigations and Disclosure Requirements) Regulations, 2015, this is to inform you that the Company’s wholly owned subsidiary, Deepak Chem Tech Limited (formerly known as Deepak Clean Tech Limited) has signed a Memorandum of Understanding with Government of Gujarat on 23rd May, 2023 with an intent to invest around Rs. 5,000 Crores in next 4 years for setting up projects to manufacture Speciality Chemicals, Phenol / Acetone and Bisphenol at Dahej / Nandesari, in the State of Gujarat.

The proposed investment is expected to generate around 1,500 direct and indirect employment opportunities.

The proposed investment will also help reduce India’s import bill. Products out of these projects have varied end use such as:

Speciality Chemicals cater to the demand of Agro Chemicals and Pharma industries.

Phenol and Acetone are used in varied end-user segments like laminates, plywood, pharma, paint, adhesives etc.

Bisphenol caters to Epoxy and Adhesives and is the most important feedstock towards Polycarbonate which has various applications across Automotive, Electronic and Consumer Goods, Defense, Medical Equipment etc.

Please take the same on your record.

Timeline of Deepak Nitrite’s Stock Movement

How the news broke

🕒 09:15:00 - Market Opens

Opening Price: Rs. 1948 per share.

The market starts its day quietly, with no signs of the storm brewing ahead.

🕒 10:25:57 - News Hits the BSE

Price at Announcement: Rs. 1944.95 per share. This will serve as our reference point to measure the stock’s upside potential.

🕒 10:29:56 - NSE Publishes the News

Price: Rs. 1956.15 per share (+0.575% in 4 minutes).
Volume Spike: From 200–300 per minute to 6,000–7,000 per minute.

The sudden activity likely signals the presence of automated trading bots, responding faster than human traders could.

🕒 10:31:00 - Reuters on TradingView

Price: Rs. 1955 per share (+0.5%).
Volume: Continues at heightened levels.

Despite the news spreading further, the price sees little movement, consolidating gains.

🕒 11:55:04 - Major Public Coverage (Business Standard)

Price: Rs. 2125 per share (+9.25% from the reference price).
Volume: Peaks in six-digit figures per minute.

For nearly 1.5 hours, the stock showed a linear price increase, offering a golden window for attentive traders to capitalize. By this point, most of the upside had already been captured.

🕒 15:30:00 - Market Closes

Closing Price: Rs. 2130.05 per share.
Final Upside: Minimal movement since the Business Standard article.

Key Takeaway: Timing is Everything

By the time news reaches the mainstream media, it’s often too late for the average investor. In this case, the majority of returns were realized within 1.5 hours of the initial announcement. This underscores the advantage of being well-informed and having the tools to act swiftly.

Further timeline

🕒 11:34:24 (May 25) - Exchange Seeks Clarification

The Exchange has sought clarification from Deepak Nitrite Ltd on May 25, 2023, with reference to Movement in Volume.

🕒 12:47:09 (May 25) - Standard Response

Further, there is no information / announcement (including pending announcement) which in our opinion may have bearing on the Price / Volume behavior in the scrip and which is required to be informed to Stock Exchanges in terms of Regulation 30 of SEBI (LODR) Regulations, 2015.

It’s still 20% high since that days closing even in this bearish market

To wrap things up, there’s a similar and even more fascinating story in the book Business Adventures: Twelve Classic Tales from the World of Wall Street. In the chapter “A Reasonable Amount of Time”, it talks about how insiders at Texas Gulf Sulphur tried to cash in on secret information before the news went public. It’s a gripping tale that ended up changing how insider trading is dealt with today. If you found this case interesting, that one’s definitely worth checking out!

Feel free to share your thoughts or insights in the comments!

Raspberry Pi Camera Live Streaming: Step-by-Step Setup

2023-10-01T00:00:00+00:00

Are you ready for some weekend tech fun? Buckle up because we’re about to embark on an exciting journey! Our mission? To create a cutting-edge Raspberry Pi-based image streaming service that brings real-time images right to your phone, no matter where you are. And the best part? It’s all about keeping things locked down tight — your feed, your privacy, and your excitement!

🔧 What You’ll Need for this Awesome Project 🔧

Gather up your tools because we’re about to embark on a tech adventure that’s as exciting as it is educational! Here’s your checklist:

🍇Raspberry Pi: The tiny computer with mighty capabilities, perfect for running small computing tasks right from the comfort of your home or lab.
📷Raspberry Pi Camera: A budget-friendly 5MP camera that captures surprisingly sharp and clear images.
☁️AWS EC2, Azure VM, or a Cloud VM Hosting Service of Your Choice: We’ll harness the power of the cloud to set up our very own Nginx server with mutual certificate authentication.

Setting up Raspberry Pi

Having any OS on Pi should be fine. We shall be using Python in this project. However, I recommend popular Linux distros like Debian or any OS based on Debian.

You can skip this step if you already have a working Pi. Or follow resources like this to flash your SD card and boot and set up Debian on the PI.

Connecting the Pi Camera

With the Pi setup complete and the Pi up and running, let’s get the PI camera on the Pi. Many resources are available on the internet to help us with this. You may follow this one.

Pipeline

In this project, we shall be creating this pipeline. A brief overview of the steps:

Python program uses the picamera2 module to periodically capture images
Encode and send the picture as an HTML file using SCP
Create an Nginx config file to serve the file
Create and install a key certificate to authenticate the client

Pipeline image

The Code

We are using Python 3 here, and picamera Module. However, picamera doesn’t work with python3. To fix that, there is this Picamera2 replacement (still in beta); we shall use that instead.

try:
    from picamera2 import Picamera2, Preview

    picam2 = Picamera2()
    preview_config = picam2.create_preview_configuration()

    picam2.configure(preview_config)

    picam2.start()
    while True:
          print("Clicking picture")
          metadata = picam2.capture_file(IMAGE_PATH)
          # Image to Base 64 string
          image_b64 = get_b64_string(IMAGE_PATH)
          # Format a boiler html with this string
          html = get_html(image_b64, datetime.now())
          # scp the file to the server
          export_html_file(html)
          sleep(SLEEP_DURATION)

except Exception as e:
    logger.exception('Failed: ' + str(e))
else:
    if picam2:
        picam2.close()

When Creating a valid HTML string from the base 64 encoded image, add <meta http-equiv=”refresh” content=”{refresh}”> to the head tag to auto-refresh the image in the browser.

Running the script in daemon mode

Once the code is up and running and sending images to the remote server, we need to ensure that it gets up and running automatically when the PI restarts.

To achieve this, let’s use systemctl. Create and paste the following code into /etc/systemd/system/picam.service file:

[Unit]
Description=PiCam process.

[Service]
Type=simple
User=<user>
Group=<user>
ExecStart=/bin/bash -c '/usr/bin/python /<project main file path>'

[Install]
WantedBy=multi-user.target

This will create a basic systemd service to run the program. To set it to auto-start on reboot, run sudo systemctl enable picam to enable it. Check sudo systemctl status picam to see if there were any errors.

On the Server

We all want our private feeds to be secure, but who wants the hassle of entering IDs and passwords every time? Not us! That’s why we’ve got a game-changer in store: SSL Mutual Authentication.

Authenticating via certificates: SSL mutual authentication setup

By default, HTTPS is a one-way authorization where the client checks if the server is indeed who he claims to be. However, it also supports mutual authentication, where not only does the server verify the client’s identity, but the client can also verify the server’s identity.

To set it up, we need to generate some certificates and keys

# Generate an RSA private key
openssl genrsa -out client.key 4096
# Generate self-signed certificate
openssl req -new -x509 -days 365 -key client.key -out client.crt

Nginx setup

Added this in the server block to enable client validation

location /<image_feed_url_slug>.html {
if ($ssl_client_verify != "SUCCESS") { return 403; }
alias /<image_path>.html;
}

# our self signed certificate
ssl_client_certificate /<path>/client.crt;

On the client machine (browsers)

Run the following to get a PKCS12 file

# Export the private key and certificate to PKCS12 file. Remember the password
openssl pkcs12 -export -inkey ./client.key -in ./client.crt -out ./client.p12

You can install this PKCS12 file in the browser’s user cert section of the certificate settings page. (Use the password when prompted)

For Android devices, if the above doesn’t work, you need to do the following additional steps (based on the answers here) to get a legacy PKCS12 file:

# Converts the PKCS12 file to a PEM file without password
openssl pkcs12 -noenc -in client.p12 -out temp_client.pem
# Converts the PEM file to legacy PKCS12 file. The -legacy option tells OpenSSL to use the legacy PKCS12 format, which is compatible with older devices.
# Add a password
openssl pkcs12 -export -legacy -in temp_client.pem > client_legacy.p12 # Add new password for android

Install this legacy PKCS12 on the Android device with the new password. Set it in VPN and app user certificate

If all went well, you must be able to see the live image feed from the camera on you browser

What Lies Ahead

Taking Your Project to the Next Level:

As you wrap up this fantastic journey of setting up your Raspberry Pi-based image streaming service, it’s time to look to the future. Here are some exciting possibilities to explore:

Enhanced Performance: Take your streaming to the next level by implementing optimizations like sending only the difference in images or transmitting images when motion is detected. These tweaks can make your service even more efficient and responsive.
Dynamic Live Feeds: You can expand your project to support video-based live feeds. Imagine having real-time video streaming from your Raspberry Pi straight to your phone. The possibilities are endless!

By delving into these further steps, you can continue to elevate your project, making it even more powerful and versatile. The world of DIY tech is yours to explore, so keep innovating, experimenting, and enjoying every moment of your tech journey. 🚀🔍📹

Income Tax Filing for Foreign Assets

2023-07-27T00:00:00+00:00

Let’s be real — filing for income tax in India can be a bit of a head-scratcher. It’s like trying to solve a puzzle where the rules keep changing, and for someone new to the whole earning-and-tax thing, it can get seriously overwhelming. You’ve got the basics down, but what about those foreign assets? The whole “I didn’t sell, why bother” strategy might not be the tax-saver it seems.

And don’t get me started on the CA route. It turns out that even they sometimes draw a blank regarding the nitty-gritty details.

And here’s the kicker — mishandling those foreign assets can trigger notices, bringing you face-to-face with the Black Money Act. Massive penalties lurk and the prospect of jail time is not just a distant threat.

Wait, there’s more: India and the US have agreements that allow them to exchange details about their residents’ assets. So, thinking your overseas investments are flying under the radar? Think again!

People around have been getting away with not disclosing foreign assets. The luck factor may be in play here, and you may be living on borrowed time. The department can issue notices anytime, and there is no time limitation.

Let’s unravel the tax web together, ditch the confusion, and arm you with the know-how to breeze through this financial maze. Because in the world of taxes, knowledge is definitely power! 💪💰🔍

There’s not much information on the internet available on how to fill details regarding foreign assets. This blog is my efforts on consolidating on whatever I could find from the internet and some CAs around.

Disclaimer

The following Medium blog is my perspective, and I might be wrong. The discussion is focused on stocks held in United States by Indian Residents. Things may change if you hold stocks elsewhere. Things may also change drastically with time and the details here can turn outright incorrect for that time period.

Please do your due diligence before using any information here. Your case might be different than what I discuss. Consulting your tax advisor is highly recommended.

Concept

Before we start, let’s clear a few things:

Conversion rate: We should use RBI’s reference rate or SBI’s telegraphic transfer buying rate (TTBR) for any USD-INR conversion. I am referring to this rule I found on income tax. If it is a holiday, we should take the TTBR of the immediately preceding working day. Pick the table from the TTBR pdf with “to be used as a reference” mentioned. However, SBI doesn’t provide historical data, so we may use these sources to get that:

sbi_forex_rates/csv_files/SBI_REFERENCE_RATES_USD.csv

This project downloads and stores the daily SBI forex rates in a CSV file enabling you to access historical rates…

skbly7/sbi-tt-rates-historical

Historical SBI TT rates since 02 July 2020. These are one of important rates required for ITR purposes and not made…

Forex Card Rates

All the content on this site is published in good faith and for general information purpose only. Mksco does not make…
For filing Foreign Asset-related schedules, you can opt for ITR2 or ITR3. Choose ITR3 if you have some business income.
Rounding errors: Please use all the required decimals for the calculations. Round up the final number while filing.

Relevant Schedules when having Foreign Assets are:

Schedule Foreign Assets (FA)

The department expects any Indian resident to declare whatever foreign asset one might have. The following excuses will not work:

It is RSU, and the company has already deducted the required tax and declared it in Form 16.
I didn’t sell anything.
I didn’t move the money to India.

This schedule is only for declaration and bookkeeping by the department. United States issues proofs in January to December cycle (current year). Hence, we should declare assets till December.

This schedule has a lot of tables for declaring the different kinds of assets. I will only discuss cases when we have bought stocks or have RSUs from the company.

Stock declaration

Use Table A3 for Schedule FA. Some people use Table D (which is simple) as well but Table A3 is recommended.

In Table A3, we should fill each individual holding separately. It must be acceptable to club all the buy trade for a day in a single entry but across multiple days should have its separate entry. Currently, we will have to enter the details manually every time. This is cumbersome, but it’s good to do it this way. For RSUs, what this means is that all unsold/ partially sold vesting should be added separately.

Sample Table A3 will have the below essential fields, which I have explained using this example.

I bought 100 units on 21 Dec 2021 @ $200 and sold 40 units from Jan 2022 to Dec 2022 @ $300. For FY 2022–2023, in foreign assets, I should fill the following:

Date of acquiring: Date of purchase (or vesting date): 21st Dec 2021
Initial value of the investment: Amount paid on the purchase. = 100 * 200 * TTBR conversion rate on the date of investment. If you have these units at a discount via ESPP (Employee Stock Purchase Program), you should probably add the undiscounted fair market value here.
Peak value: Max value of the investment during the holding period between Jan to Dec = 100 * max value of the stock during the period * TTBR of the date when that stock has max value. In practice, people likely take the peak value for the entire year (not just the holding period maximum).
Closing value: The closing value of investment till 31st Dec = 60 * closing value of stock on 31st Dec * TTBR
Total gross amount paid/credited with respect to the holding during the period: Calculate the total dividend and interest for this particular unit from Jan to Dec. The conversion rate probably will be calculated based on the dividend payment date. Take the TTBR of the last date of the immediately preceding month. This is the most computational work of the entire process.
Total gross proceeds from sale or redemption of investment during the period: Total value of stock sold during the Jan to Dec period from that lot = 40 * 300 * TTBR on the date of selling of stocks

Every data should be proportionally reduced for the next year to match 60 unit holding.

Broker balance declaration

The broker balance (presumably because of the dividends and interest accumulated) is also a foreign asset that must be declared. Choose Table A2 for filing the details. For the closing balance, use the dollar balance as of 31st December and multiply with the TTBR on that day to find the amount in INR. Report details of deposits transferred into the account directly or through dividends. If we sold some units during the current year, please consider that too for peak and closing balance calculation.

Even if the account has no balance, it’s good to declare it as such.

Schedule Foreign Source Income (FSI)

Unlike Schedule FA, all details here should be according to Financial Year in India, i.e., from April to March. Add details for any income received or receivable even if the money is yet to move to India. Also, do consider advance tax in mind. If you don’t pay the taxes appropriately in the time frame of advance taxes, the advance tax penalty may also be levied.

Tax treatment of dividends

Ordinarily, Indian dividends are subject to be taxed on the hands of the receiver based on his slab. The same applies to any foreign dividends received. However, the United States government already deducts 25% of the dividend as tax before distributing it. In essence, there could be a case of double taxation on the same income.

The Indian and US governments signed the Double Taxation Avoidance Agreement (DTAA) to help taxpayers with this issue. Effectively, we are still taxed the same in India. However, we request the Indian government to give us relief for the 25% already paid which is added as a credit in the final tax liability calculation.

So effectively, if you are in the 30% slab, you still need to pay 5% more.

Relevant fields here:

Income from outside India (included in Part B-Ti): Total dividend pre-tax calculated using TTBR of the last date of the previous month.
Tax paid outside India: 25% of the total dividend withheld.
Tax payable on such income under normal provision in India: Manually calculate and add the tax based on the slab.
Relevant article of DTAA if relief claimed u/s 90 or 90A: 10, 25.

Stock Capital Gains

For the stocks, the capital gains are the (sell value in $ — buy value in $) * TTBR on the last date of the preceding month to the sell date.

Interest on broker balance

Similarly, fill interest received on broker balance here.

Schedule Tax Relief (TR)

Put the summary of tax relief claimed for taxes paid outside India here. Add the same relief amount (calculated above) again in this section.

For the Tax Relief Claimed under section (specify 90, 90A or 91), for RSUs, stocks and dividends in the US add section 90.

Form 67

Form 67 is a statutory form used by resident taxpayers in India to claim credit for foreign tax paid outside India.

Add the exact details here as well. The numbers across Form 67, Schedule FSI and Schedule TR should match. It is also advised to submit form 67 before submitting the ITR.

As understandable, this section should also be filled according to the Indian financial year.

Proof is required to be uploaded when submitting this form. We should get Form 1042-S from the broker for the dividends and interests. Form 1042-S is an information return filed by withholding agents to report amounts paid to foreign persons from the US. However, this document is only issued in the current year, January to December. (You can probably submit the broker statements for the period Jan to March)

Schedule Income from other sources (OS)

This is the schedule for adding all the additional interest and dividends received from foreign shares during the financial year: April to June.

Ideally, for all individual dividends received, you should convert the pre-tax amount to INR using TTBR of the last date of the immediately preceding month. Add the cumulative INR in the dividend section. You should give the division among various advanced tax periods for the dividend received.

Do the same thing for interest on the broker balance and add it under the “Others” row in the interest section.

Schedule Capital Gains

All capital gains (including those relating to foreign assets) should be reported here. For foreign assets, taxation is a bit different.

Capital gains on foreign assets

Any holding for more the 2 years (from vesting/ purchase date) qualifies for special tax treatment under long-term capital gains. These are taxed at 20% after indexation.

Indexation means that you will only be taxed for the gains above inflation. So, if you sold your holding in 3 years making 50% profit, and assuming inflation to be 5% annually, the inflation will be 5 * 3 = 15% (well it is more than that as the number will be compounded). So, you actually pay only 20% of (50%–15%) as tax (plus cess and surcharge 🥲)

The loss can also be adjusted with other profits as well (check the rules for that). Any charges directly linked to the transfer can also be claimed. Technically, the transfer of foreign currency-related charges should not be claimed under capital gains as it’s not linked to the transfer of assets. But it probably will not be questioned if we do.

Divide the earnings into timeframes based on advanced tax periods similar to what we did in dividends.

In addition, if your annual income exceeds 50 lakhs, you should also fill out Schedule Assets and Liabilities.

Please comment below if you feel something is incorrect or if I missed something. This blog has been an effort to consolidate the data on the internet. I will be more than happy to modify the content.

References

https://indcdn.indmoney.com/public/images/itr_filling_video.mp4
How to report your foreign income, share investment while filing ITR form?

Many individuals having investments in foreign shares, or having bank account in foreign country are required to report…
RSU of MNC, perquisite, tax, Capital gains, ITR

What are RSUs or Restricted Stock Units? What is vesting date? When are RSU taxed? What is the capital gain from…
e-Filing Form 67 in New Income Tax Portal

DISCLAIMER: The following Medium Article is only for knowledge purpose and to allow taxpayers to gain more insight into…

Jio vs TCP

2022-11-15T00:00:00+00:00

🚀 Diving into the Network Abyss: Unraveling Jio’s 64kbps Terminal Limit from a TCP Perspective 🌐💻

Hold on to your virtual hats, tech enthusiasts! We’re about to embark on a journey that goes beyond the surface of India’s telecom giant, Reliance Jio. We’re delving into the mysterious realm of Jio’s 64kbps terminal limit, not to dissect Jio but to understand the intricate dance it performs in the world of TCP.

Sure, Jio is a behemoth in the telecom world, with its 5G plans and a game-changing impact on data costs that brought the nation online at lightning speed. But today, we’re not just talking about Jio; we’re on a mission to decode the enigma behind the 64kbps terminal limit, unravelling the threads that make TCP tick in this innovative landscape.

Forget the usual chatter about DAU/MAU numbers; we’re here to explore the techy side of Jio’s prowess. So buckle up, tech explorers, as we venture into the heart of Jio’s network intricacies, where IPv6 adoption reigns supreme at a staggering 92.5%. Let’s push the boundaries of understanding and discover what makes Jio’s TCP perspective truly fascinating! 🚀🔍📡

Src

TCP is arguably the most used internet protocol for data movement from the user’s device to the cloud (HTTP/3 aside). At the moment, HTTP/3 adoption hasn’t been widespread. Even the support for that in Nginx hasn’t been merged.

TCP is a reliable and connection-oriented protocol. It tries its best to get the data from one application on the internet to another. Being so important, it is supported by the whole public internet. TCP has essential features built into it. It takes care that the receiver can handle the data the sender will deliver. It also considers the network bandwidth in how it deals with congestion. During heavy load or artificial traffic shaping by on-route devices, TCP packets can be dropped. However, TCP employs various congestion protocols on either end of the connection to deal with them most efficiently while keeping the network congestion in mind. You can find more details here. The following experiment demonstrates TCP in action on a Jio ISP.

The Experiment

I have a target text file of 1.4 MB (14,45,257 bytes) on my remote server. I am fetching that file using SCP (which uses TCP at the transport layer).

Jio continues to provide data at 64Kbps speed after the unlimited daily usage.

I am trying to Scp this file to and from the server post the daily usage at 64kbps. Note that 64Kbps is 8 KBps.

Results

Time vs RTT for download test from the client

From the data above, the RTT usually is about 205 ms, which gives maximum in-flight bytes to be around 1640 bytes for that duration (barely more than the regular 1500 bytes limit for Ethernet frames)

As expected, there will be packet drops somewhere.

To understand the details below, do note that the server’s public IP is 20.203.40.255, and the client’ Jio’s external IP is 47.11.211.102. (I don’t mind sharing these details! Go ahead!)

From the server capture, just before the drops,

Server Packet logs

On receiving the ACKs from the client (number 1413), the server sends several data packets simultaneously (2782*5 in 1415–1418). This is the TCP congestion protocol in action, expanding the congestion window (CWND). Notice that these individual packets are bigger than possible to pass undropped. However, the MTU for the immediate network on both ends is 1500 and looking at the jumps in the IP identity header, this is probably TCP sending more packets of at most MSS size because of the availability of a higher window.

From the client side,

Client packet logs

After sending the ACKs for earlier packets (49–50), we see that it is starting to receive those simultaneous packets. It sends an ACK on 56. And again on 59. Post that, there is no new packet till 3.21s (about 220ms). This indicates that some of the later simultaneously sent packets were dropped. It received packet 60 but not the immediate next ones it was expecting to receive. It sends an ACK back (61) acknowledging this packet but saying there was a packet in between that was not received. Subsequently, the server only sends single TCP frames with the client’s request packets.

This is a duplicate ACK received as the 1428th packet for the server. Henceforth it sends the requested packet again. Notice that these duplicate ACK TCP packets would have dropped the server side CWND and hence must have slowed down in streaming the packets anticipating network congestion.

Server packet logs continued.

Notice the decrease of the bytes in flight. The server then only sends the requested packet segments (which Jio possibly has dropped).

Post packet 1511, it started accelerating again (TCP congestion control at work) but stopped again at 1519.

Checking the throughput on Wireshark. This is Jio not letting more than 64000bits pass in a second

Wireshark Window scaling graph:

Server Packet capture TCP Window Scaling on Wireshark

The green line representing the Receive window size stays around the initial negotiated 64128 throughout, while the blue line representing the bytes outstanding is forced to come down to MSS because that’s around what is possible to pass through at the permitted speeds.

Interestingly, there is no ICMP packet recorded on either side notifying of packet drops in the network. It was all about duplicate TCP ACKs and TCP Out-of-Order packets and TCP Previous segment not being received.

The upload traffic test results were similar.

These cycles of increasing bytes in flight and retransmission drop occur continuously throughout the run. This is how TCP copes with packet drops on the network. This is TCP vs Jio in action.

Networking when you google

2022-05-16T00:00:00+00:00

I took a lecture aimed at explaining what happens when you google something. It was more like how a website is accessed from networking perspective. This lecture touches:

The Transport layer protocols
- TCP
- UDP
Explains what happens when we search for a website on the browser
- DNS
- Packet exchange on the network

Check this out here

Realtime Results Livestream

2021-06-19T00:00:00+00:00

Get ready for a quick tech thrill! Last month, we dove into a time-sensitive project for the West Bengal state elections - a 5+ hour YouTube Livestream featuring real-time data in a dazzling design. This blog spills the secrets behind the scenes, from coding marathons to unexpected triumphs. Fasten your seatbelts - it’s a tech spectacle you won’t want to miss! 💻🌐🎥

Now, let’s get real. We had zero experience with a programmatically defined dynamic Livestream. None. Nada. Zilch. But hey, where’s the fun in sticking to the tried and tested?

The Stream Design

I already had the idea to add a dynamically generated state map based on the party’s colour with a lead on the constituency. This is what we (I and Karthik(Karthik Iyer)) envisioned our final design to be:

A state map dynamically generated. Maps for phases and district as well.
State-wise overall results of the 5 states on the poll
Constituency-wise leads. (Party-Coalition and candidate leads)
Comparison with the past result
Dot-semicircle pie chart (for visualizing majority)
Projected total seats
Notification panel (Recent flips in constituency leads)
Time, Name & Logo

~ Alas, we dropped some of these in favour of some of the important ones.

The Pipeline Design

We designed individual components in this pipeline in parallel, using basic classes (to be subclassed later) in the backend. Every component was designed to reduce the coupling amongst them so that a crash on the scraping module (or even the backend in general) won’t affect the frontend logic. This was mainly done because of the uncertainty in the pipeline with respect to the scraper and the procured data. The main reason was that this script was to be developed on D-day only (for obvious reasons).

Pipeline diagram

Major components in the pipeline were:

Backend: Actual scraping and database logic with map rendering logic exposed through Flask APIs
Frontend: React app with Material UI components for elegant and easy design.
Selenium Browser & pyvirtualdisplay: A chrome instance with the frontend running in the background (to replace the need for a secondary display).
OBS studio: Tool to capture and stream the virtual display to YouTube.

This pipeline by design has got a number of resource-intensive services running. Henceforth, we focussed on reducing the back-end processing by always leveraging cached data for the API calls. We prefetched and stored all that we could, like candidate details, for example. We only scraped constituency-wise per candidate vote count on the go. The frontend periodically queries the backend to check for updates. And the backend independently updates the database and maps. A periodic refresh task triggered the scraper, which then updated the database’s state continuously. This essentially made the backend logic independent of the API calls.

We developed events-based callbacks triggered by events, like when the leading candidate in a constituency changed or when the overall ranking changed.

Map Rendering

The map renderer registered itself with these callbacks, through which it received just the updates in data. We created metadata for rendering the maps beforehand. We were able to find a constituency level map of West Bengal in the MS Excel Map format through sheer luck. It contained all constituencies, paired with an integer value and a chart map highlighting individual constituencies based on these values. We wrote an Excel macro that changed the value paired with each constituency, highlighted it yellow and took a screenshot. All automated. Generating metadata after this was simple. A simple python script that cropped the screenshots to the appropriate dimensions, found all the pixels coloured yellow and stored the pixel coordinates in a simple text file.

Map rendering individual constituency

When the server starts, it loads all those pixel coordinates into an array. When the update callback is triggered, we fetch these coordinates only for those constituencies which actually changed and re-render just these pixels. We used Pillow to render the images. Once updated, the image is saved onto the filesystem. When the frontend requests for a map, it is fetched from that cached map from the filesystem. To ensure thread-safety for simultaneous read-update to the filesystem, we came up with this solution: to preload paths to all the map images in an array, use paths from this array while saving or reading an image use a thread lock on the path strings.

Results

All set and done, the live stream worked perfectly for more than 5 hours.

Running this heavy pipeline on a single device did take a toll on the performance. It was mostly around dropping frame rates, but it was unnoticeable on the stream for the most part. Below are the details for the environment running the pipeline:

Ubuntu 20.04 LTS Linux with 5.8.0 kernel on Predator Helios 300 with Intel corei7 9th gen processor, 16 GB RAM & 6 GB NVIDIA GeForce GTX 1660 Ti Graphics card.
Flask and React app was running on debug mode and OBS studio

Before closing the stream, I recorded these system statistics.

The performance was getting increasingly poorer as we approached towards the end of the stream.

htop

top

nvidia-smi

Note that

The backend was running through the Pycharm’s terminal.
python display.py is the process running pyvirtualdisplay + selenium process.
Google-Chrome was also running parallel.

Other processes are rather self-explanatory and/or mostly comes preinstalled with standard Ubuntu desktop installation.

I scanned all my broadband peers

2021-04-22T00:00:00+00:00

Hold onto your internet connection because things are about to get interesting! I recently went on a scanning spree, unveiling the mysteries of my broadband peers and the results? Mind-blowing! 📡🔍 Ready to dive into the fascinating world of broadband revelations? Let’s roll! 🚀💻

My internet broadband provider is one of the popular Internet service providers in this region. I have been their customer for quite some time now. Though comments and reviews may point otherwise, I am satisfied by their service in my region with close to promised speeds mostly.

They use NAT and provide static IPs under the 10.0.0.0/8 private block, and if I am not wrong, this same subdomain is used in West Bengal’s operational area. As of this moment, they don’t block any cross-customer communication. This makes it possible to scan the entire network. And as I have mentioned later, this may be a security concern.

The masscan configuration used for the scan was:

masscan 10.0.0.0/8 -p80,23,443,21,22,25,3389,110,445,139,143,53,135,3306,8080,1723,111,995,993,5900,1025,587,8888,199,1720,465,548,113,81,6001,10000,514 -Pn -n - rate=1000 -oL 10.masscan

This ensures the following:

Masscan helps find open IP:Port pairs faster
Scan the top few TCP ports (32 in this case)
Keep the scan slow, so it doesn’t cause any network issues, and I don’t wake up anyone at the ISP.

Below, I have mentioned some interesting facts on this scan and related work.

Out of the ports mentioned, the distribution of count vs port number was

Port count
 194
138
 75
 59
15
10
9
 3
2
1

(excluding port 25 & 53)

A lot of routers

Quite fewer Ip cameras

And other login pages of things I don’t know about

Takeaways

Vendors providing these routers & IP cameras should prompt the users to change the default password upon 1st login. A good number of people never care to change the default passwords of these devices after working installation. At the same time, default settings for these devices should restrict access to a select few IPs to prevent unauthorized access over the WAN.

IP cameras should particularly actively provide patches to discovered vulnerabilities and prompt users to update their devices for the same. These vulnerabilities are regularly detected and patched, but it does take time for these consumer devices to get patched.

Network operators should actively block cross-customer communication because consumers usually don’t care or don’t have the knowledge to be safe against it. Therefore some responsibility should be borned by the ISPs.

Actions

I have raised a service ticket and mailed them explaining the issue. The support representative has promised to forward this to the senior authority.

Inspiration

The following blog by captnemo was the inspiration for this blog:

I scanned all of ACT Bangalore customers, and the results aren’t surprising

I scanned all of ACT Bangalore customers, and the results aren’t surprising

DNS and Privacy

2021-02-15T00:00:00+00:00

Ever felt like your internet journey is a little too exposed? As you surf anonymously, websites slyly snatch your public IP address, and your ISP lurks, peering into your every click. It’s the perfect stage for unwanted surveillance, with the potential to tweak your routing behaviour and block access to your favourite sites. Enter the unsung hero: DNS.

In this blog, we’re delving into the world of online invisibility. I’ll spill the beans on the latest tech tricks that can cloak your browsing habits from the prying eyes of ISPs, with a special focus on the wizardry of DNS. Ready to discover the secrets to a more private online experience? Let’s dive in!🚀🔍

Domain Name System

DNS is the phonebook of the Internet. It’s a global database of information about domain names. In simple terms, it helps convert medium.com into 104.17.31.52 which computer networks actually understands and can route your request to.

Source

A simple fully qualified domain name looks like en.wikipedia.org.. DNS follows a hierarchical structure. A dot separates each level. At the top of this structure is the root servers which specifies details about these top-level domains (TLDs) for example, org or com. These also act as the trust anchors (as explained later) and the first level while querying DNS records. More on querying can be found here.

Dig: DNS lookup tool in Linux

A is the most basic type of DNS record which contains IP addresses to which this domain should point to. There are a few other records too for other purposes. These records have a Time to Live TTL field associated with them describing how long to cache them before requesting a new one. Caching would improve response time for the subsequent queries and reduce load over DNS servers.

For the uncached plain text query to 8.8.8.8 for medium.com, Wireshark recorded round trip time (RTT) of 62.43 ms and chrome recorded 62.72 ms. That’s 4.8% of basic page load time. Subsequent queries were cached by chrome itself to reduce load time.

DNS generally works on UDP over port 53. TCP over the same port can also be used, but UDP is the prime choice. UDP, if you know, is unreliable but it is fast. For a simple query, where TCP establishes a connection using 3-way handshake before the actual query, UDP could complete it in a single round trip. (DNS query is the first thing your browser does when visiting a website). Use of TCP also needs more resources on both sides and more header data compared to UDP.

DNS clients generally switch to TCP when the DNS payload is more than 512 bytes. It may continue on UDP if the devices and the network supports larger DNS packets over UDP. The 512-byte UDP payload size is a dependency on IPv4. From RFC,

All hosts must be prepared to accept datagrams of up to 576 octets (whether they arrive whole or in fragments). It is recommended that hosts only send datagrams larger than 576 octets if they have assurance that the destination is prepared to accept the larger datagrams.

The number 576 is selected to allow a reasonable sized data block to be transmitted in addition to the required header information. For example, this size allows a data block of 512 octets plus 64 header octets to fit in a datagram. The maximal internet header is 60 octets, and a typical internet header is 20 octets, allowing a margin for headers of higher level protocols.

This is why there are precisely 13 DNS root servers originally: 13 domain names and 13 IPv4 addresses fit nicely into a single UDP packet.

A plaintext UDP DNS response for medium.com

The original DNS did not include any security details; instead, it was designed to be a scalable distributed key-value system. As you know, DNS query and response are transferred as plain text UDP packets and are susceptible to be forged and manipulated while on the wire. There were no checks on the authenticity of the data in the original implementation. Users could be easily redirected to a fake website using DNS spoofing.

DNS Spoofing Source

Keeping in mind how easy it is to manipulate and spy on DNS data, certain extensions and improvements have been specified and used. Some of them are:

DNS Security Extensions (DNSSEC)

The DNS Security Extensions (DNSSEC) is specifications for securing the DNS data. It provides data integrity and authentication to security-aware resolvers and applications through cryptographic digital signatures while ensuring backward compatibility with insecure DNS. These digital signatures are included in those resource records. DNSSEC does not provide data confidentiality; in particular, all DNSSEC responses are authenticated but not encrypted.

DNS Validation Source

DNSSEC creates a parent-child train of trust that travels all the way up to the root server. The root keeps the hash of DNSKEY for the authoritative top-level domain (e.g., .com) nameservers (in the DS record). These (e.g., .com) TLDs keep hash for DNSKEY for their subdomains (e.g., example.com). All the record sets are signed by their own DNSKEY and stored as a new record at each level. This is a lot like the chain of trust used to validate TLS/SSL certificates, except that, rather than many trusted root certificates; there is one trusted root key managed by the DNS root maintainer IANA.

Formally published in 2005, it is still far from mainstream adoption - one of the reasons is network operators who prefer stability to complexity (for a good reason).

DNS over TLS (DoT)

Source

DNS over TLS, or DoT, is described in RFC 7858. The protocol uses port 853 (though port can depend on the implementation). It is based on TLS (also known as SSL). This is a standard to encrypt the communication between the user’s device and the DNS resolver. To an external device, only the targetted resolver will be known. This prevents spoofing and tracking of the DNS messages.

DNS over TLS incurs additional latency at session startup. It also requires an additional state (memory) and increased processing (CPU). Compared to UDP, it requires an extra RTT of latency to establish a TCP connection. The TLS handshake adds another two RTTs of latency. Clients and servers should support connection keepalive (reuse) and out-of-order processing to amortize connection setup costs.

DNS over HTTPS (DoH)

DNS over HTTPS, or DoH, is described in RFC 8484. The protocol uses the standard HTTPS port 443, thus making DNS traffic have the same semantics with regular Web HTTPS traffic from web browsers, for the external world. It is also based on TLS (as it is based on HTTPS), thus making things similar to DoT.

Based on HTTP, the DNS request-response is replaced with HTTP request-response messages over GET or POST method. Templates can be defined for making these requests and responses, and the standard status codes can have similar meanings.

HTTP/2 is the minimum recommended version of HTTP for use with DoH. The messages in classic UDP-based DNS are inherently unordered and have low overhead. A competitive HTTP transport needs to support reordering, parallelism, priority, and header compression to achieve similar performance. Those features were introduced to HTTP in HTTP/2. Earlier versions of HTTP can convey the semantic requirements of DoH but may result in inferior performance. Even other HTTP/2 features like Server push is supported.

DoT and DoH are very similar. But there a debate on what is better.

From a network security standpoint, DoT is arguably better. It gives network administrators the ability to monitor and block DNS queries, based on port, which is important for identifying and stopping malicious traffic.

DoH is applied at the application layer. In contrast, DoT is applied directly at the transport layer. As such, DoH wrt DoT has.

More coding required
More libraries required
Packet sizes are more extensive than DoT.
Ever-so-slightly higher latency

Neither DoT nor DoH is perfect DNS encryption solutions at this moment. More work still needs to be done, like enabling 0-RTT for all DNS-over-TLS and DNS-over-HTTPS implementations.

There have been more number of DoT deployments than DoH. Support for DoT has been added to Android since Android Pie. systemd-resolved in Linux already has an option to turn this on. Recently, Firefox has added an option to use DoH with its browser. Chrome has been catching up too. For browsers, it is easier to use DoH as they are already handling HTTPS traffic. Google and Cloudflare have provided their open DoH resolvers.

Below is a beautiful and detailed explanation of encrypted DNS:

A cartoon intro to DNS over HTTPS - Mozilla Hacks - the Web developer blog

DNS sinkhole and Pi-hole

A sinkhole is a DNS provider that can provide a false result for particular DNS queries. This may be because of a compromised server or server being configured to block access to specific domains (for parental control, advertisement filtering, or preventing botnet attacks).

Pi-hole logo

Pi-Hole is an excellent, free, and open-sourced DNS-Sinkhole with support on major platforms (from Pi as in raspberry pi and hole in DNS-Sinkhole). It comes with a standard list of websites (including google analytics!) to block, to prevent DNS queries to harmful and unwanted domains, and block 3rd party advertisements, all together saving bandwidth. It has a pretty web interface for configuration. Domains can easily be added or removed from the blocklist. It provided faster DNS queries by optimizing DNS caching with respect to native DNS implementations. It comes with a configurable DHCP server to automatically allocate DNS servers to the clients. It can easily be installed on a home or corporate network.

Pi-hole admin panel

As evident from this image, 21.5% of all my traffic was blocked (primarily advertisements).

Check out my image for a container for Pi-hole based DNS Sinkhole with Cloudflared DoH setup. It has a handy bash function too.

dev-ritik/Pihole-Docker

This repo has the Docker environment for Pihole (DNS sinkhole and DHCP Server) + Cloudflared (DNS over HTTPS proxy). I…

For a locally uncached DNS request for medium.com in this setup, Wireshark recorded RTT of 91.47 ms and Chrome recorded 91.85 ms. That’s 9.1% of basic page load time and a 46.4% increase in DNS query time in DNS over HTTPS with Pi-hole (as expected).

At the same time, when the results were cached by Pi-hole locally (and not by Chrome),

it had an RTT of 0.145 ms in Wireshark and 0.39 ms on Chrome. This makes sense for having a DNS resolver on the local network.

Plain Text Server Name Indication

With the setup above, one can be sure that no on-path device can detect what website someone is visiting based on the DNS traffic. There are still few other exploitable methods to find that out. The easiest of them will be the Server Name Indication (SNI).

A web server can host multiple ‘virtual’ servers on a single IP. These servers may belong to different DNS domains. Reverse proxy technologies like Nginx may be used to distribute incoming traffic to an appropriate server based on the domain it contains. This would be easy in the case of HTTP traffic (as it contains the requested domain in the URL), but for encrypted HTTPS traffic, things are quite different.

TLS Handshake is performed before the actual HTTP(S) communication. For this, the client needs to validate the server’s certificate to establish trust (explained here). In a TLS handshake, the server can send a common certificate of all the virtual hosted domains to the client, generating the session key if the requested domain was within the certificate. But this is not a scalable approach. The more widely used technique is where the client will specify what virtual domain it wants to talk to, with the request.

Client Hello (for example.com)

This is sent in the Server Name Indication field in the Client Hello. As you can see, the server name is sent in plain text and can easily be used to spy on a device. Anyone between you and the server can see this field. This is the first message of the TLS Handshake; hence, no key has been agreed upon yet to encrypt this field.

One approach to encrypt this field is by encrypting Client Hello. Encrypted Server Name Indication(ESNI) works by adding a Public Key to the DNS records. A client can easily fetch this key (while querying for the IP address) and use this to encrypt the SNI field. Since only the server will have the corresponding private key, only it can get the original field and no one else.

Certificate as plaintext in TLS v1.1 Server Hello

This is added as an extension to TLS v1.3. Older versions of TLS even expose the server’s certificates, so set the minimum acceptable version to v1.3. Details of this can be found here. As of now, there are very few deployments of ESNI. You can enable this feature in the configs (about:config) in Firefox. Check out this website and find out how secure are you against anyone spying on your online activity.

Cloudflare ESNI Checker | Cloudflare

When you browse websites, there are several points where your privacy could be compromised, such as by your ISP or the…

Firefox v85 has recently stopped supporting ESNI towards an improved Encrypted Client Hello (ECH) TLS extension. The basic idea is to encrypt the entire client rather than just the SNI field (Client Hello does contain other sensitive information). This would still be supported using DNS by adding keys and other metadata to DNS records. The draft RFC explains its working as

When a client wants to establish a TLS session with the backend server, it constructs its ClientHello as usual (we will refer to this as the ClientHelloInner message) and then encrypts this message using the public key of the ECH configuration. It then constructs a new ClientHello (ClientHelloOuter) with innocuous values for sensitive extensions, e.g., SNI, ALPN, etc., and with an “encrypted_client_hello” extension, which this document defines. The extension’s payload carries the encrypted ClientHelloInner and specifies the ECH configuration used for encryption. Finally, it sends ClientHelloOuter to the server.

The backend server (for example, Cloudflare) could be serving multiple domains. It acts as a relay decrypting the Client Hello and forwarding that to the original server. Cloudflare has put up a really nice blog post on this.

As Firefox explains here, they have dropped the support for ESNI, and the support for ECH is under active development. As such, for now, downgrading Firefox to Extended Support Release v78.7.1esr seems to be a good option.

Even now, anyone can guess what website you are visiting based on the IP address of the server you are visiting. In most cases, results would be quite accurate as well. Fortunately, as more and more websites are hosted on services like Cloudflare, they share common IP addresses. This will then become more difficult to find out. For any other cases, it’s best to route your traffic through some other servers by using a VPN or using a Tor browser.

Network Access Control

2020-12-13T00:00:00+00:00

Dive into my CSN-502 presentation delivered on 31.10.2020, where we unravel the intricacies of Network Access Control, with a special focus on Wireless Protocols. If you’re keen to delve deeper, check out the references for more details. This serves as your quick intro to the topic; the ppt is attached below. 🚀💻

Network Access Control

You must have heard that open and coffee shop wifis are insecure, and one must proceed with cautions. Now think about large corporates or campus networks. There’s a ton of stuff involved here which are private within the network and must be protected via access control and prevented from eavesdropping. Remember, a network is not just used for connecting to the internet. A single malicious device or some malicious programs in some innocent device can bring massive troubles to the network providers. All other stuff related to secure access management applies here as well.

So broadly, it is the job of the Network Access Control System to proactively detect and authorize the devices and put certain access control in place. It should also provide a mechanism to deal with malicious programs and actors acting with bad faith from within or outside the network.

Network Access Control ppt

Broadly any general NAC system should have at least these functionalities:

Node Detection: This indeed is the first step to find out if there’s a new device connected to the network. The system can detect the traffic from protocols like ARP, which are essential when registering in a network or from explicit NAC register messages.
Authentication: Authenticating the device or the user trying to access the network.
Assessment: A secure network must assess a device before giving it access to the system’s resources for potential threats.
Authorization: NAC must use some policy to provide the user with access to the network’s resources based on the user’s/device’s identification and assessment.
Quarantine: NAC can isolate non-compliant devices into a separate network where it doesn’t possess a threat to the network and can still access specific safe resources.
Remediation: Quarantined devices should be allowed to join or get access to system resources upon undergoing remedies like downloading security patches or firewall setup.
Monitor: The devices admitted to the network should be continuously monitored for some policy violation upon which it can be Quarantined, or the access can be dropped and actions can be taken.

An NAC sysem generally has these components:

Client: These are the devices trying to access the network. These are of 2 types: Agent-based Client, where an agent (a software) is already installed on the client. This client then communicates with the policy servers to make the client compliant with network security parameters like installing anti-virus software. Agent-less Clients are made to install an agent for joining the network. If it cannot, its traffic is monitored for threat management.
Enforcement Points: These are points in the network where action can be taken for a device. These are generally the points with which the client communicates to access the network. Example: Switches, Routers, VPN.
Policy Servers: As the name suggests, these servers are responsible for policy enforcements like Access, Authorization, and Accounting. These servers are responsible for collecting assessment data and ensuring compliance.
Quarantine Network: This is a network (as defined earlier) for the non-compliant devices. They are allowed limited or no access to network resources and remediation servers.
Remediation Servers: These servers are responsible for checking Quarantined devices for their compliances with network standards. These can provide with required security software to those devices for the purpose.

Extensible Authentication Protocol

EAP (RFC 3748, 5247) is an authentication framework, not a specific authentication mechanism, used in network and internet connection for providing the transport and negotiation of authentication methods via EAP methods. EAP typically runs directly over data link layers without requiring IP. EAP is not a wire protocol; instead, it only defines the information from the interface and the formats. Each protocol that uses EAP defines a way to encapsulate by the user EAP messages within that protocol’s messages. It has been applied to a wide variety of networks and protocols like wired networks IEEE-802.1X, Internet Key Exchange Protocol version 2 (IKEv2) RFC4306, and wireless networks such as [IEEE-802.11] and IEEE-802.16e.

EAP defines 4 (generally 3) entities for authenticating a device. Some of these entities can be and are usually handled by the same device in a setup. These are:

Supplicant: This refers to the device seeking admission into the network.
Authenticator: The entity which relays and initiates EAP authentication. It may not know specifics of the actual wire protocol used.
Authentication Server: This is the place where the actual authentication takes place.
EAP server: Its job is to exchange EAP messages with the supplicant via authenticator. The authentication server then uses these messages. It is often handled by other entities.

EAP Authentication Process

This image describes EAP over LAN (802.1X)

The authenticator sends a Request to authenticate the peer (code 1). The Request has a Type field to indicate what is being requested.
The peer sends a Response packet (code 2) in reply to a valid Request. As with the Request packet, the Response packet contains a Type field, which corresponds to the Type field of the Request.
The authenticator sends an additional Request packet, and the peer replies with a Response. The sequence of Requests and Responses continues as long as needed.
The conversation continues until the authenticator cannot authenticate the peer in which case the authenticator implementation MUST transmit an EAP Failure (Code 4). Alternatively, the authentication conversation can continue until the authenticator determines that successful authentication has occurred, in which case the authenticator MUST transmit an EAP Success (Code 3).

Security Claims

EAP is used in wireless and wired LANs. In these situations, an attacker can gain access to links over which EAP packets are transmitted. As an example, one can easily detect all the traffic going in a wireless network without even authenticating into the network. As such, there are several security concerns which EAP defines and the EAP methods should take care of. These can be found here.

EAP Methods Example

EAP-MD5: Here, the server sends a random challenge to the client. The client forms an MD5 hash of the user’s password and the challenge and sends the result back to the server. The server then validates the MD5 hash using the known correct plaintext password from the user database. Hence, It offers minimal security, is vulnerable to dictionary attacks, it only provides authentication of the EAP peer to the EAP server but not mutual authentication, hence is vulnerable to MITM.
EAP-TLS: Transport Layer Security (TLS) provides for mutual authentication, integrity-protected cipher suite negotiation, and key exchange between two endpoints. EAP-TLS is still considered one of the most secure EAP standards available. This has also been used in NoAuth cases where only server authentication is done. This would allow for situations, much like HTTPS, where a wireless hotspot allows free access and does not authenticate clients. Still, clients wish to use encryption and potentially authenticate the wireless hotspot. With a client-side certificate, however, a compromised password is not enough to break into EAP-TLS enabled systems because the intruder still needs to have the client-side certificate.
EAP-PSK: EAP method for mutual authentication and session key derivation using a Pre-Shared Key (PSK). The Pre-Shared Key refers to a key or secret that needs to be derived and shared by the parties by some mechanism before the EAP-PSK conversation takes place (Take this as a simple passphrase). EAP-PSK is designed for authentication over insecure networks such as IEEE 802.11(standards for Wireless LAN (WLAN) & Mesh (Wi-Fi certification)). EAP-PSK is distinct from the Pre-shared Key authentication mode used in Wi-Fi Protected Access(WPA) and should not be confused with it.

Implementations

Wi-Fi (Wireless Fidelity) History

Wired Equivalent Privacy (WEP) is a security algorithm for IEEE 802.11 wireless networks. Introduced as part of the original 802.11 standard ratified in 1997, its intention was to provide data confidentiality comparable to that of a traditional wired network.

Later, to address WEP security issues, the 802.11 working group adopted the 802.1X standard for authentication, authorization and key management. IEEE 802.1X is an IEEE Standard for port-based Network Access Control (PNAC). It provides an authentication mechanism to devices wishing to attach to a LAN or WLAN. At the same time, IEEE formed a Task Group “I” to develop 802.11i standard, with a purpose to produce a detailed specification to enhance the security features for wireless LANs dramatically.

In 2003 the Wi-Fi Alliance announced that WEP had been superseded by Wi-Fi Protected Access (WPA). They extracted the key features from 802.11i to establish WPA to satisfy the immediate needs of the wireless industry. WPA became available in 2003 based on a subset of a draft of 802.11i. IEEE 802.11i describes EAP for wireless LANs.

However, WPA has shown significant vulnerabilities and was later superseded by WPA2, which is based on IEEE 802.11i standard ratified in June 2004.

In January 2018, Wi-Fi Alliance announced the release of WPA3 with several security improvements over WPA2.

WEP

WEP was ratified as a Wi-Fi security standard in September of 1999. WEP uses the stream cipher RC4 for confidentiality and the CRC-32 checksum for integrity. WEP generally uses Pre-Shared Key (PSK); however, it uses rudimentary variations of the same key to encrypt all packets . These weak encryption keys are based on the underlying RC4 crypto algorithm. This is not very safe, a large sample of encrypted packets using the same key tends to create an easy key recovery target for hackers. Furthermore, to change a WEP key requires an IT administrator to update each client machine manually. As a result, changing pre-shared keys regularly to safeguard against key recovery is a highly unfeasible task to scale for large deployments.

Systems that rely on WEP should be upgraded or, if security upgrades are not an option, replaced.

WPA

As discussed above, WiFi Protected Access was formally adopted in 2003, a year before WEP was officially retired.

WPA keys use 256-bit keys. Its most common configuration is called the WPA-PSK or WiFi Protected Access Pre-Shared Key. WPA included message integrity checks (to determine if an attacker had captured or altered packets passed between the access point and client) and the Temporal Key Integrity Protocol (TKIP). TKIP employs a per-packet key system that was radically more secure than the fixed key system used by WEP. Although TKIP uses the same base encryption algorithm, RC4 as WEP, the way it selects and changes keys resolves many of the issues surrounding WEP.

The primary improvement in WPA is the per-session encryption key. Every time a station associates, a new encryption key is generated based on some per-session random numbers and the media access control (MAC) addresses of the station and the access point. WPA sounds like a major improvement, and it is — if it’s used correctly.

WPA with pre-shared keys can be cracked if IT management unwisely uses straightforward passwords. However, this is not a weakness in WPA security, but rather a potential result of poor password management. An attacker grabs a few packets at the time a legitimate station joins the wireless network and then can take those packets and recover the PSK used.

WPA used with 802.1X authentication (WPA-Enterprise) offers a secure, per-session encryption key that is not vulnerable to any casual attack. This security comes with a cost because 802.1X authentication requires a significant infrastructure including 802.1X-compliant RADIUS server with a digital certificate, and client software for every user that supports 802.1X and whichever authentication mode they use.

WPA2

WiFi Protected Access II, as discussed above, was ratified in 2004, and replaced WPA. WPA2 implements the mandatory elements of IEEE 802.11i. In particular, it includes mandatory support for CCMP(Counter Cipher Mode with Block Chaining Message Authentication Code Protocol), an AES based encryption mode as a replacement for TKIP.

Working

four-way handshake

The four-way handshake is the process of exchanging 4 messages between an Access Point (AP) (authenticator) and the client device (supplicant) to generate some encryption keys which can be used to encrypt actual data sent over Wireless medium.

While Enterprise version use 802.1x exchanges (as discussed above) before handshake, personal directly begins handshake using Pairwise Master key (PMK) (generated from PSK).

The Master Session Key (MSK) is the first key which is generated either from 802.1X/EAP (Enterprise) or derived from PSK authentication (Personal). Pairwise master is key generated from this master session key. In case of WPA2/PSK when device authenticates with access point the PMK is simply derived from PSK.

Here, with the four way handshake, the client is able to generate a Pairwise Transit Key (PTK) and Group Temporal Key (GTK). All unicast traffic will be encrypted with PTK and all multicast and broadcast traffic will be encrypted via GTK.

PTK = PRF (PMK + Anonce + SNonce + Mac (AA)+ Mac (SA))

For PTK, PRF is a pseudo-random function, Anonce is supplied by the AP in the 1st message. SNonce is generated by the client. Mac address of the AP and supplicant is known already. Hence PTK can be generated. GTK is generated by the AP and shared with the client in the 3rd message of the handshake.

For detailed explaination, follow this post.

Issues

The ability to crack the WPA2-Personal passphrase with brute-force attacks: basically guessing the password over and over until a match is found, is a critical vulnerability of WPA2. Making the problem worse, once hackers captured the right data from the airwaves, they could perform these password-guessing attempts off-site, making it more practical for them. Four-way handshake is susceptible to offline dictionary-based attacks, especially when short passwords under 16 characters are employed. Once cracked, they could then decrypt any data they captured before or after the cracking.

Another major vulnerability of WPA2-Personal, particularly on business networks, is that a user with the passphrase could snoop on another user’s network traffic and perform attacks. This is more evident in open networks. Although the enterprise mode of WPA/WPA2 protects against user-to-user snooping, it requires a RADIUS server or cloud service to deploy the enterprise mode.

KRACK attack

The Key Reinstallation Attack (KRACK) is a direct attack on the WPA2 protocol and not in individual products or implementations.

Essentially, KRACK undermines the four-way handshake, allowing a hacker to intercept and manipulate the creation of new encryption keys within the secure connection process. The adversary tricks a victim into reinstalling an already-in-use key. When the victim reinstalls the key, associated parameters such as nonce and receive packet number (i.e. replay counter) are reset to their initial value.

During the four way handshake, if the 3rd message is received again, the client will reinstall the same encryption key, and thereby reset the incremental transmit packet number (nonce) and receive replay counter used by the encryption protocol. By forcing nonce reuse in this manner, the encryption protocol can be attacked, e.g., packets can be replayed, decrypted, and/or forged. Adversaries can use this attack to decrypt packets sent by clients. As a result, the same encryption key is used with nonce values that have already been used in the past. In turn, this causes all encryption protocols of WPA2 to reuse keystream when encrypting packets. In case a message that reuses keystream has known content, it becomes trivial to derive the used keystream. This keystream can then be used to decrypt messages with the same nonce. When there is no known content, it is harder to decrypt packets, although still possible in several cases. In practice, finding packets with known content is not a problem, so it should be assumed that any packet can be decrypted.

WPA 3

In 2018, the Wi-Fi Alliance announced the next iteration called WPA3 to replace the WPA2, adding several security enhancements and features while overcoming the security vulnerabilities of the WPA2. It takes security to the next level by making wireless networks future ready. It uses the more powerful and robust encryption by AES with the GCMP (Galois/Counter Mode Protocol) and replaces the PSK with the more reliable and secure handshake mechanism called Simultaneous Authentication of Equals (SAE). SAE, also known as the Dragonfly Key Exchange Protocol, is a more secure method of key exchange that addresses the KRACK vulnerability. Specifically, it is resistant to offline decryption attacks through the provision of “forward secrecy.” Forward secrecy stops an attacker decrypting a previously recorded internet connection, even if they know the WPA3 password.

Have a look at this post for details.

References

Kishanganj-2020

2020-11-06T00:00:00+00:00

Ever heard of the Kishanganj Vidhan Sabha seat? Probably not. But hey, let’s shed some light on the names vying for your vote in the Bihar Legislature election 2020. I get it; politics isn’t everyone’s cup of tea, but consider this your crash course in “Know Your Choice!” These nuggets are just bits and pieces from my internet scavenger hunt. I’m just here to spark your political curiosity! 🕵️‍♀️🗂️

Bihar Election 2020

Political parties

The main parties in Bihar relevant to Kishanganj (or at least those I will be considering here) are:

Indian National Congress (INC): is headed by Madan Mohan Jha in Bihar.
Bharatiya Janta Party (BJP) is headed by Sanjay Jaiswal in Bihar.
Janta Dal (United) (JD(U)) is headed by CM Nitish Kumar.
All India Majlis-e-Ittehadul Muslimeen (AIMIM) Akhtarul Iman is said to be the president of AIMIM, which recently got its first seat in Bihar from Kishanganj.

54 Kishanganj Vidhan Sabha

54 Kishanganj Vidhan Sabha

Candidates

MD. Qamrul Hoda (AIMIM). He won the constituency in 2019 bye-election when the sitting Congress MLA won the MP seat in 2019 Lok Sabha elections.
Ijaharul Husain (INC). After the backlash in 2019, when Congress decided to give its ticket to the candidate from the family and lost badly, the party has decided to give it to a new face, a party worker.
Sweety Singh (BJP): 3 times securing 2nd position, she has an ever-increasing vote per cent and has a good chance of being first this time.
Tasiruddin (Indp): One of the independent candidate with a relatively poor performance in the recent election.

Below are the details of these major candidates. These data are largely taken from the affidavit submitted by them for contesting past state elections. These affidavit though is a good source of information but at times these are poorly filled.

I couldn’t add a table here, so here’s their overall data

Candidates general details

Immovable assets include the market value of all the assets inherited.

Candidates expenditure is recorded according to the Election Commission’s Guidelines. These are put in place to reduce the bias created because of the money power of the candidates. Candidates are required to maintain a register of the same and this register is compared with a shadow register maintained by the officials. This data is sourced from the same. Note that Qamrul Hoda was absent for the first term and presented a common report in the 2nd term. (However absent and not answering in a requested period is considered really bad). Third term’s report is yet to be uploaded.

Individual profile

Except for the new Congress face, the 3 of them have been contesting state elections. the data below is taken from the affidavit of the candidates from their past elections and the election results available in the Election Commission Website.

Note that all financial data here is the sum of self, spouse and dependents.

Past elections in Kishanganj

Note that 0 indicates that the party did not have a candidate under its flags, probably because of a coalition.

Below is the Turnout (the count of people who actually voted) and Electors (the total number of voters registered in the area).

Note that the dip in 2010 for Vidhan Sabha can be associated with the delimitation of Parliamentary and Assembly constituencies Order, 2008

Source

The cover image is taken from Wikipedia.
Kishanganj Vidhan Sabha Map is taken from Election Commission’s records.
The tables are generated by me using the data made available by the Election Commission and Kishanganj’s page.
https://eci.gov.in/statistical-report/statistical-reports/
Election

Bihar Assembly Election 2020-Nomination Paper Checklist Tender from District Election Office Kishanganj Bihar Assembly…
Some of the data is also sourced from various Wikipedia articles.

~ Happy Voting!

Trust-TLS-SSL & HTTPS

2020-05-17T00:00:00+00:00

In our last rendezvous with HTTP, we dove into its evolution for better performance. But here’s the plot twist — HTTP alone is about as secure as a paper umbrella in a storm. Yet, that little lock🔐 in the corner of your browser assures you of safety in the vast and sometimes treacherous realm of the internet. Trust me; my server’s access logs tell tales of the wild digital west!

So, what’s the secret sauce behind securing HTTP traffic? What’s the story behind that reassuring lock? How do we establish trust in this cyber jungle? And most importantly, is this Medium article genuinely what it claims to be?

Join me on this journey as we unravel the mysteries of HTTP security. I’ll be your guide, using real-world examples to break down complex concepts without drowning you in technical jargon. Let’s embark on a tech adventure where security meets simplicity! 🚀🔐🌐

Prerequisites

A bit of knowledge of network systems, particularly the TCP, and cryptography will be enough to understand the content. Though, an interest in the topic should suffice anyway!.

HTTPS??

Hypertext Transfer Protocol Secure (HTTPS) is the secure version HTTP. HTTP, as the name suggests, sends data packets as hypertext. Anyone on the internet can easily read these, or hackers can impersonate your server upon using compromised routers like cafes and other public routers or easily tracked and modified by your ISPs. As such, critical information like banking details should not be handled without HTTPS. Even if you don’t host any sensitive data on your server, it’s pretty recommended shifting to HTTPS. (You don’t want an intermediary to modify your content like pushing their advertisements with it without your permission and your website down rated and marked insecure by popular browsers.)

HTTPS helps keep these malicious elements out by using what is called TLS/SSL Handshake, which happens before the actual data is transferred. The crux of this handshake is that your server and your client generate a shared session key using asymmetric cryptography, unique to the world, and then encrypt all the traffic to random data packets. Yes these packets can still be seen and modified by those sitting in the middle, or even the request can be forwarded to a malicious server, but your client software would know that something is wrong and alert the user to switch to a more secure network. How would they know you say? Well, that’s what this blog is all about.

The actual messages are formatted and structured as determined by HTTP. It’s just that they are encrypted to garbage while transmitting and again decrypted by the intended recipient and none else. All HTTP data MUST be sent as TLS application data. By default, HTTP uses port 80 and HTTPS uses port 443. Using HTTPS, only the source & destination IP address, port and domain (SNI) are public. The URL, query parameters, headers, body, and cookies (which often contain identity information about the user) are all encrypted. The amount of data and the duration of communication can also be determined.

Making HTTPS calls using OpenSSL

OpenSSL is a command-line cryptography tool relating to SSL/TLS protocols and related cryptographic functions. As such, it can be easily used for making HTTPS calls. On your terminal,

openssl s_client -crlf -connect example.com:443
GET / HTTP/1.1
Host: example.com

Simple Example

Run the following code to fetch example.com using python. Comments have been added for your help.

Example of a simple HTTPS request

Capturing HTTPS and HTTP Packets

There are a handful of tools, particularly Wireshark, which helps us analyse network packets actually exchanged. Let’s use a similar tool tcpdump to sniff and analyse these packets going to example.com (93.184.216.34). It can sniff all the packets on the target interface and help in analysing the traffic.

HTTP: Let’s use cURL to make an HTTP request. Run this command on the terminal to capture packets:

tcpdump -A -n -s 0 host 3.220.159.10 and port http

Now, make an HTTP cURL request from another

curl --location 'http://postman-echo.com/post' --data-raw 'This is a client secret, not to be revealed!'

This would print the TCP packet in ASCII. If you like, you can get in hex as well. (You can interpret that packet using TCP segment structure as well). Anyways, here you can see among other things, src and dest address & port. What you can also see is the first three messages of the TCP handshake (SYN, SYN+ACK, SYN). Following that you can see the entire HTTP request:

E....B@.@.. ..+#...
.D.P..EQ.t.............
.U....[.POST /post HTTP/1.1
Host: postman-echo.com
User-Agent: curl/7.58.0
Accept: */*
Content-Length: 44
Content-Type: application/x-www-form-urlencoded

This is a client secret, not to be revealed!

And after the server ack, the server reply as well:

E.. .a@…{….
..+#.P.D.t….F….n…….
..]T.U..HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Apr 2020 17:59:17 GMT
ETag: W/”189-BP2UptWJTIcEngOzrSrBUV1O57Y”
Server: nginx
set-cookie: sails.sid=s%3Anxu4rLgsmTVDxMM8s-9m8UvJ-RniOXlj.x4Jd80JEc2ldtbGd9KVKcnOP%2Bkspz0dYKduOgl4%2FXL0; Path=/; HttpOnly
Vary: Accept-Encoding
Content-Length: 393
Connection: keep-alive
{“args”:{},”data”:””,”files”:{},”form”:{“This is a client secret, not to be revealed!”:””},”headers”:{“x-forwarded-proto”:”https”,”host”:”postman-echo.com”,”content-length”:”44",”accept”:”*/*”,”content-type”:”application/x-www-form-urlencoded”,”user-agent”:”curl/7.58.0",”x-forwarded-port”:”80"},”json”:{“This is a client secret, not to be revealed!”:””},”url”:”https://postman-echo.com/post"}

Following an ack by the client, you can see TCP Connection Termination as well.

The dump has other interesting things as well for your interpretation.

Take away here is that a middleman can sniff your complete HTTP request-response easily.

HTTPS: Use Openssl for HTTPS and make the same request. Trace it using:

tcpdump -A -n -s 0 host 3.220.159.10 and port https

You will see a lot of messages exchanged, from both TCP Handshake and TLS Handshake. But you won’t find any more data than the TCP headers. HTTP data is encrypted to arbitrary bits.

SSL

Secure Sockets Layer (SSL), more commonly called Transport Layer Security (TLS), is a standard security protocol, commonly used by HTTPS, for securing the communication. Data is transferred over the public internet wherein it could easily be spoofed or sniffed if sent as plain text. SSL ensures that this transfer is encrypted using symmetric and asymmetric cryptography. Thus the data actually transferred is garbage to anyone else on the internet(planet) other than their intended recipient. (Even though they might never have met each other and are sitting on the other sides of the planet.)

This is done by what is called SSL Handshake and Certificate trust verification. This SSL protocol dictates how both parties should interact to ensure trust and authentication. All this is carried using what is called an SSL Certificate. This certificate packs a lot of data within it together with a public key. Please note that SSL is not limited to HTTPS. Other protocols like SMTP (for sending emails) are secured as well like SMTPS.

SSL Handshakes

To generate that unique session key for encryption, both parties need to agree on particulars of the conversation. So this Handshake is that pre-conversation agreement. During the Handshake, the following things happen:

Both parties agree on the protocol version.
Decide on the cypher suites to use
Prove the identity of the server (and the client if required)
Decide on the symmetric common session key for both parties

courtesy: https://techcommunity.microsoft.com/

Again, it all starts with a Socket based connection (A TCP handshake). The server is waiting on accept, i.e. waiting to accept a new TCP Socket connection and the client (browser) will connect soon. What happens during a socket connection (TCP Connection) is handled at the transport layer level. During the SSL Handshake, this is generally what happens:

Client Hello: The client sends a request for a secure connection with the server. It requests that the web server identify itself. It also includes other fields for initiating the agreement like supported versions and cyphers.
Server Hello: The server replies with its SSL certificate proving it’s identity. Note that the certificate also contains its public key. It also replies with some of its fields like selected version and cypher. It may ask for a client certificate request.
Authentication & ClientKeyExchange: The client verifies the certificate received, through which it confirms that the server is indeed who it claims to be and the public key does genuinely belong to the server and not forged by someone on the public internet. (Again it’s all Cryptography!) The client, upon trusting the certificate and the public key, generates an asymmetric pre-master secret and sends it encrypted using the server’s public key. (Intelligent only that server can decrypt it using its Private key.)
Private key: The server decrypts the pre-master secret using its private key.
Session keys: Both client and server perform a series of steps depending upon the cypher and computes the session key. Hence, by the laws of cryptographic algorithms, they have shared a symmetric key which only these 2 devices on the planet claim to have and no other on the public internet (sorry MITM).
Finish: The server and the client acknowledge each other by sending a “finished” message that is encrypted with the session key!. The secure connection is thus established, and their communication is encrypted now.

The exact steps may be slightly different based upon the cyphers and the version of the protocol used.

Note that in this form of connection, the client certificate is not required. SSL verification of clients is also possible and required under some conditions wherein the server requires the client to verify as well! Well browsers don’t fit in those conditions.

Certificates

The SSL Certificate establishes the identity of an entity on the Internet. It is like a small document of trust wherein the entity records its public key and organisational details etc. The credibility of these certificates is established on the chain of trust anchored to the trusted authorities on the Internet using cryptographic wizardry!

Let’s get our hands dirty on these certificates!. There’s a lot to grab in them and let’s try to pocket some of them. It’s effortless to get the certificates of a web service. Using openssl is one such method. For medium.com, you can see the certificate using this command:

openssl s_client -showcerts -connect medium.com:443 </dev/null

Most popular browsers let you get these certificates as well.

Starting with a sample certificate!. In your browser, click on that lock symbol on the search bar and download Medium’s certificate, it used to negotiate an SSL connection with your browser. This is how it looks now:

This looks like a lot of base64 encoded gibberish but saves this in a file medium.crt and run the command,

openssl x509 -in medium.crt -text -noout

You can see all the details it has within itself. Among other things, it contains,

Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA

Indicating that DigiCert SHA2 Extended Validation Server CA issued medium this certificate and

Subject: businessCategory = Private Organization, jurisdictionC = US, jurisdictionST = Delaware, serialNumber = 5010624, C = US, ST = California, L = San Francisco, O = A Medium Corporation, CN = medium.com

is the description of the entity(medium) it is issued to. Now if you download DigiCert SHA2 Extended Validation Server CA‘s certificate and run similar command, this is what you get: Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA

Indicating that this itself was issued by DigiCert High Assurance EV Root CA. You can see a chain like trust-building! Upon downloading this one, you will see that issuer is the same as the one to whom it is issued to,

Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
Validity
Not Before: Nov 10 00:00:00 2006 GMT
Not After : Nov 10 00:00:00 2031 GMT
Subject: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA

Medium certificate chain

So should we trust DigiCert High Assurance EV Root CA and by extension medium!? Well yes! They are the trust anchors (Certifying Authorities) of the internet trust network, and mostly every SSL software keeps a copy of these certificates or gets it through support libraries. They have to comply with the set of security guidelines established by these Softwares (like Browsers). As can be seen here, certificates are generally signed by other certificates forming the certificate chain. This chain terminates at a self-signed authoritative certificate provided by these root Certifying Authorities (CA)s like Digicert Root CA. There are cryptographic algorithms to ensure that if certificates are indeed signed by the issuer (discussed later).

Also, if you are following along using openssl, you must have noted that medium doesn’t actually send this root certificate. The browser itself, in fact, adds it. The server may not send the root CA certificate if it wants to. (RFC Reference).

You can also verify the certificate chain using openssl. As explained in the picture above, Issuer (Digicert CA) signs the certificate containing the public key of the requester (Medium). It uses it’s own private key to generate a cryptographic signed hash of the data provided with the request and sends that in the certificate. Now cryptography says that only Digicert’s public key can decrypt it, which is available with its own certificate. So let’s get

Digicert pubkey: To get the included Digicert’s public key from its certificate use

openssl x509 -pubkey -noout -in DigiCert\ SHA2\ Extended\ Validation\ Server\ CA.crt > DigiCert.pub

Medium signature from the certificate: You can extract the Digicert signed hash in the medium’s certificate and convert it into binary:

openssl x509 -in medium.crt -text -noout -certopt ca_default -certopt no_validity -certopt no_serial -certopt no_subject -certopt no_extensions -certopt no_signame | grep -v ‘Signature Algorithm’ | tr -d ‘[:space:]:’ | xxd -r -p > medium_signature.bin

Cert body: To get the body of the cert which was actually hashed,

openssl asn1parse -in medium.crt -strparse 4 -out cert-body.bin

Finally, run the command (sha256 in this case),

openssl dgst -sha256 -verify DigiCert.pub -signature medium_signature.bin cert-body.bin
>>>> Verified OK

This shows that Digicert CA actually signed the medium’s cert and everything in medium’s cert (including medium’s public key) is actually genuine if Digicert’s certificate is genuine. This certificate chain of trust is the backbone of trust on the internet. More details on this can be found here. This was just signature verification, but other things like validity and revoking trust check also comes into play.

Where to look for the certificates I trust? Softwares like your OS comes with its list of trusted anchors.

For Linux, the file /etc/ssl/certs/ca-certificates.crt contains the concatenated list of trusted CA’s certificates which defines the base trust for your OS and other software which use that file as default.
requests library in python has this file wherein it looks for the root CA certs. You can see that it depends on the certifi package. This package contains updated root CA certificates for major trust anchors. (Like if you see their PyPI page, ‘Python package for providing Mozilla’s CA Bundle’). Certifi comes with its list of certs when downloaded, available here.
In python urllib3, it says here that, ‘Unless otherwise specified urllib3 will try to load the default system certificate stores. The most reliable cross-platform method is to use the certifi package which provides Mozilla’s root certificate bundle’.
For chrome, go to chrome://settings/certificates onto the Authorities tab, and search for DigiCert, you will get DigiCert High Assurance EV Root CA there. If you compare these two, they are actually the same! Hence Chrome establishes trust for this certificate for you! And hence you see that green lock there on the search bar!. You can get it on the Firefox as well!.

Look how chrome complained when I denied chrome to use that certificate to verify medium! XD

Manually removing trust for Digicert Root CA.

Not Secure Warning on the search bar

NET::ERR_CERT_AUTHORITY_INVALID

Let me explain what just happened. During SSL handshake, Medium didn’t provide all the 3 certificates! It just provided its own certificate, cryptographically signed by DigiCert SHA2 Extended Validation Server CA bundled with the latter’s certificate signed by DigiCert High Assurance EV Root CA. But since Chrome now doesn’t trust this Root CA, it cannot guarantee the user if it is indeed medium. It may be any malicious website pretending to be medium and proceeding further is probably unsafe.

Let me describe another case. Consider that the server doesn’t send its certificate chain and rather send the leaf certificate (signed by an intermediate authority and not directly by your trusted root CA). What happens then!. Well, it all depends upon the implementation of the certificate verification program. Certificates contain fields (along with Issuer DN (Distinguished Name) mentioned above) which help the program to fetch that intermediate certificate from other sources. This may be utilised to fetch and build the chain and hence build trust. Chrome does this automatically for you, but firefox probably gives a warning. But for libraries like python request, this isn’t the case. You will certainly get verification errors in this case! Here, a temporary solution will be to add intermediate certificates yourself as trusted (for the time being).

Simple Example

To get the simplest SSL based communication, you would need a certificate for the server, which can be verified by our demo client. This is explained beautifully here.

For experimenting with different types of certificates, have a look at this website.

How to get a certificate for my website?

Self-Signed Certificates!

As you must have guessed by now, you can indeed generate your own self-signed certificate and use that to establish yourself as the trust anchor for your closed testing environment. But that won’t work in production. Your users won’t trust your signed certificate. Instead, browsers and search engines will demote your website. Indeed your communication will be secure though! (assuming that it is with your server and not any other impersonated server!)

You would obviously try to get certificates from established trust anchors on the internet. Now there are tons of CA on the internet both paid and free ones. I personally use Let’s Encrypt for my websites (not a paid promotion!). It’s free and easy to use. To get a certificate for your website’s domain from Let’s Encrypt, you have to demonstrate control over the domain. Assuming that you are hosting your service on some virtual servers like AWS yourself, it basically boils down to your server listening to the IP address attached to your domain and you having SSH access into the system. It comes with a handy tool called certbot. It can automate certificate issuance and installation with no downtime. For my website, I have automated the process using these shell commands:

$ wget https://dl.eff.org/certbot-auto; chmod a+x certbot-auto
$ sudo ./certbot-auto certonly -- non-interactive -- email <email> -- agree-tos -- standalone -- domains <domain> -- keep-until-expiring -- debug
$ sudo ln -sf /etc/letsencrypt/live/<domain> /etc/letsencrypt/live/cert

This generates a cert and private key for your website and stores the Let’s encrypt intermediate cert as well for the trust chain at /etc/letsencrypt/live/cert/ . Now you could easily add their location to Nginx or Apache(httpd) server config file or use them with the simple HTTPS example mentioned above.

Conclusion

🎉 Phew! That was a content-packed ride! 🚀 Unraveled the mysteries of TLS, didn’t we? Dive into the intriguing world of cryptography and explore more with my examples. Play, learn, and enjoy the tech playground. Ready for hands-on fun? Check out the repository for a real-life implementation based on what we’ve explored! 💡💻

dev-ritik/Secure-file-transfer

A demonstration for secure file transfer within an organization using key-chain validation. The files are secured using asymmetric encryption…

HTTP & WebSockets

2020-05-17T00:00:00+00:00

In our last tech rendezvous, we delved into the realm of Simple Servers and got cosy with sockets-based communication - the unsung heroes behind the scenes. But let’s face it, when it comes to writing servers today, it’s the dynamic duo of HTTP and HTTPS taking centre stage!

Picture this: the entire digital cosmos, with terabytes of data zipping around every second to the gadgets in our hands - it’s mostly all orchestrated by the magic of HTTP and WebSocket. But here’s the kicker - how do they tango with sockets? How is HTTP evolving, and is WebSocket just a fancy cousin of the classic socket?

Hold on tight as we demystify these web wonders! I’ll be your guide, unravelling the tech tapestry with real-world examples and steering clear of unnecessary tech mumbo-jumbo. Let’s embark on a tech exploration where complexity meets simplicity! 🚀🌐💡

Prerequisites

A little knowledge of network systems with interest in the topic should be just excellent. Have a look at my last blog for some basic concepts.

Introduction to HTTP and its versions

Once a socket-based connection is made, the client and the server can indeed communicate with each other. If it were for your inhouse application, any form of communication standards would suffice. But, when it comes to standardizing a protocol for communication between the browsers and the servers, it ought to be simple and extensible. As you might know, HTTP plays that role. It is the underlying application-level protocol which defines this communication.

Whenever a browser wants a web page, it sends an HTTP request to get that page. The server responds according to this protocol and sends the page and an appropriate HTTP status code which can be interpreted by the browser as the standard server’s response. Its versatility and usefulness are reflected on its extensive use even now (since the 1990s). Today, a wide range of applications (almost just about anything you can think of) are deployed over HTTP.

HTTP is a request/response protocol. The client sends a request to the server in the form of a request method, URI, and protocol version, followed by a message containing request modifiers, client information, and possible body content over the connection. The server responds with a status line, including the message’s protocol version and a success or error code, followed by a message containing server information, entity metainformation, and possible entity-body content.

Features

HTTP by itself is stateless. This means each request is executed independently, without any knowledge of the requests that were executed before it, i.e. the server does not keep any state between two requests. Even though servers use cookies today to store state, HTTP remains stateless
The applications can easily modify it by adding custom headers, which the client and the server agrees upon.
It is reliable (ensures the message is transferred or generates an error).
HTTP/1.x, it is pretty easy to implement, something which I demonstrated in my socket blog.

Versions

Take the simplest example of HTML/0.9 (the initially proposed version). As simple as making a TCP Socket connection and sending one-line ASCII request

GET /index.html

to the socket will get you the response hypertext message from the server socket

<HTML>
  <Body>
   <h1>
     Hello from server!
   </h1>
  </Body>
</HTML>

and the connection will be closed.

HTTP has had several revisions since. HTTP/1.0 modified HTTP/0.9 and added features like status code headers, version (appended in the 1st line).

Standardised in HTTP/1.1, it is the most widely used version right now. It introduced connection reuse, among other things. Opening and closing a connection is a significant overhead in a TCP connection. With the introduction of SSL/TLS Handshake, the number of message exchanges for establishing a connection has gone significantly up, increasing the load on HTTP servers apart from causing congestion on the Internet and the apparent delay. By persisting a connection, memory & CPU time is conserved as well. These warm existing connections overcome TCP slow start congestion control protocol as well. Hence, it’s wasteful if used for just a single request-response exchange connection.

Using persistent connections, multiple HTTP requests could be sent in the same connection (connection is ‘keepalive’ by default). This allowed connection to be reused, but the problem remained that newer requests had to wait for the response of the existing one in a particular connection(a strict FIFO from the client-side).

Courtesy

It introduced request pipelining as well wherein, multiple requests could be sent instead of waiting for the response of the first one to arrive, on the same connection. Modern-day webpages request tons of other static assets as well upon loading. Pipelining them on the same connection makes sense as well. But this increases the load of buffering and processing on the server’s end. As you can see in the picture, the server can only send the response to a request in a FIFO manner. Multiplexing was still not introduced for helping with multiple parallel responses. This has some other severe implications in TCP based connection as well. Therefore pipelining is in marginal use today.

In such a scenario, browsers have limited choice for handling tons of static resources put into displaying today’s web pages. They have to make a balance between the number of parallel connections and FIFO delay. (Throwing unlimited connection may lead to unintentional D-DoS). In practice, browsers tend to open around 6 connections per host, but there is a good discussion around that. From RFC 2616, “Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy”.

Waterfall for the same domain using HTTP/1.1

You can verify this using the inspect-network tool in the (chrome) browser. Notice how only 6 parallel connections are made (to the same host) and other requests are waiting for the response to the earlier ones.

There is yet another problem with HTTP/1.x. When considering tons of HTTP calls for a single page rendering, everything matters. If you notice, we are exchanging a bunch of metadata including headers and cookies(which by the stateless nature needs to be sent with all of your requests) which fundamentally increase the total payload for the transport layer. These are sent as plain text without compression. Run the command:

curl --trace-ascii - 'http://postman-echo.com/get?foo1=bar1&foo2=bar2'

You can see that the header to data byte ratio in the request is 103:0, whereas in response is 352:227. Headers clearly overshadow the actual HTTP payload.

Come HTTP/2.0, many of these problems were addressed. Essentially it adds a new intermediate binary framing layer between the HTTP/1.x syntax and the underlying transport protocol, without fundamentally modifying it. With the introduction of multiplexing transfers, multiple requests could be sent simultaneously in the same TCP connection with each request being largely independent of others in the form of separate streams. These streams are just a bidirectional flow of HTTP/2 frames, identifiable by assigned integers. Multiple streams can be initiated and used at the same time within the same connection, thus establishing multiplexing.

Courtesy

The basic protocol unit in HTTP/2 is a frame. There are quite a few types of frame, including HEADERS and DATA frame. The format of frames is well defined in RFC 7540. Along with other stuff, these frames carry a stream identifier, necessary while multiplexing wherein these frames of different streams can be interleaved and then reassembled. In the HEADERS frame, headers are compressed and serialised into a header block using HPACK compression. Often these header names and values are the same or similar. It uses Huffman encoding and static and dynamic reference tables for this purpose. Same headers among consecutive frames can be simply referenced using the dynamic table and duplicates can be dropped. You can get a feeling of how this would be done and feel the compression gain as well. Header frames may also contain a weight field, assigning priority to the requests. This prioritisation will help in getting crucial assets like CSS and blocking JS faster, thus improving performance.

With the introduction of server push, servers can push additional resources to the client which it knows the client would request for. The server can send these in additional multiple responses (in new push streams) for the single request. This indeed helps in saving request latencies in RTT (Round Trip Time). For example, the server may push all the relevant assets when it receives a request for the index page, which the client can cache.

Standardised in 2015, HTTP/2 has seen wide adoption, particularly with high traffic servers. The protocol requires no change from the web application but only an upgrade to the browser and the server. The application side API remains the same with the same semantics. Because the data is transferred using binary instead of text, tcpdump cannot be used to read the data directly. As such it can run with TLS or as cleartext TCP but is generally used with HTTPS.

Check the latest implementations of HTTP/2 here. As a fact, Medium is using H2 as of now!

Implementation

Waterfall for the same host using H2

Clearly HTTP/2 is vastly different wrt HTTP/1.1 (from the image far up), you can see that the time for the first two items remained almost similar (owing to the way the page is loaded), but after that 15 more assets are requested at the same time. You can see that in HTTP/1.1, requests were queued in 6 parallel TCP connections waiting for their turn, but here, all of them are requested at once (with set priorities) without waiting. Notice how the only JS request (3rd one) took less (626 ms) to load from the HTTP/1.1 (734 ms). This can be attributed to high priority being assigned to the blocking JS assets by chrome. As for the increased size of the green bar (time to first byte) for others, it is because HTTP/2 prioritises critical JS assets for a better experience, so other requests have to wait for a while. Also, do note that the finish time is 2.30s in HTTP/1.1 vs 2.12s in H2

With Server push

For the image above, I pushed CSS and JS assets (2nd and 3rd, respectively) with the index page (1st request). Notice how it makes no network call for these resources as gets it from the cached files (pushed from server). This is a significant boost in performance and user experience wrt other images above.

Let’s move to the more recent and under-development, HTTP/3 (HTTP over QUIC(Quick UDP Internet Connections)). Though much support is not available, popular browsers and servers are slowly adopting HTTP/3. QUIC, unlike other revisions, is somewhat radical. The whole concept is to shift the entire protocol from TCP to UDP! Yes, you read it right.

Owing to some of the limitations of the underlying TCP at the transport layer, it became crucial to think of modifying that protocol. But then modifying TCP would have required changes in the entire internet infrastructure. Therefore, this new transport layer protocol is built upon UDP and inherits properties from TCP with modifications as needed.

Courtesy

To count a few limitations of TCP, wrt HTTP/2.0:

Connection establishment latency: For a general TCP secure connection, it requires 2–3 RTT in handshakes whereas it can be done in 0–1 RTT in QUIC. It merges connection and encryption into a single handshake. If the client has talked to a given server before, it can start sending data without any round trip!

Courtesy

Encryption: QUIC comes with encryption built into it and negotiates the terms in its only Handshake (TLS 1.3). Connections are encrypted from the beginning.
Stream control: In HTTP/2, wherein multiple streams utilise a shared connection, the underlying TCP has no idea of these streams. Consequently, the transport layer values them as consecutive payload sent by the application layer. This can cause some limitation such as Head of Line Blocking wherein all the streams can be blocked until a previously on flight stream frame has not been received. This is essentially handled by bringing the pre-stream control to the underlying QUIC protocol.
Connection migration: Switching networks in HTTP/2 would mean inflight packets being dropped and the need to establish a new connection (because a socket is identified using IP address port pair). This would be handled in QUIC using a 64-bit connection ID. This will help a server identify a client even when the client changes its IP address (like switching to Wi-Fi from a cellular network).

QUIC streams, like HTTP/2 streams, share the same connection. Additional control over independent streams provides additional advantages. Streams can now arrive out of order among each other and are not affected by network packet loss of another stream. However, this has to lead the HPACK algorithm to be modified into QPACK for HTTP/3, because this out of order delivery can cause some additional issues. QPACK uses separate unidirectional streams to modify and track header table state. The new HTTP/3 is built over this protocol. HTTP/3 streams simply run over QUIC streams. The semantics has not changed and is yet transparent to the application.

Much of the work is yet to be finalised, and various implementations spring up similar to the draft proposal. Check the latest implementations of QUIC HTTP/3 here.

Is WebSockets the same as Sockets?

WebSocket is a communication protocol that allows clients(like browsers) and servers to communicate on a full-duplex channel over a long time over TCP Sockets. Being based on a persistent TCP Socket connection, any of the client or the server can send data at any time until the connection is broken from either of the parties. Think about building a chat application. Pinging Server continuously for checking for new messages would be a waste of bandwidth and resources. The server should be able to respond by itself when it gets new data to send. This also differentiates WebSockets from HTTP persistent connections.

WebSockets and Network Sockets may sound similar but are quite different. While the latter is just a generic file descriptor to the file where the network writes the received data packets the former is a protocol built upon TCP Sockets. Network Socket, in general, is independent of the transport protocol(TCP or UDP).

This protocol requires a handshake over the established TCP Socket connection. The Handshake is based on simple HTTP request-response and an upgrade header and other related fields.

The handshake from the client looks as follows:

GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

The handshake from the server looks as follows:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Protocol: chat

If everything is fine, the connection is upgraded to a WebSocket connection after which any party can send data as required. Thereafter, the same connection is used for exchanging any further message. Various other control frames like PING, PONG, CLOSE are used to check/ close the connection

There are various libraries available to abstract the working of WebSockets.

Conclusion

🌟 Nailed the Basics! 🚀 Now, brace yourself for the exciting part – the promise of HTTP. Curious about the security dance? Dive into my next blog on HTTPS and TLS-SSL. The adventure continues! 💻🔒

Trust! TLS-SSL and HTTPS

How is the HTTP traffic secured? What does that lock indicate? How do we establish trust? How can I say that indeed medium.com…

See you there!

Simple Servers

2019-12-25T00:00:00+00:00

Buckle up for a swift journey! This blog is your ticket to setting up the most basic server on your computer. We’ll blend theory with easy-to-follow code explanations, breaking down the magic of socket-based servers. Get ready to unravel the secrets of the simplest sockets in no time! 🌐💡

Prerequisites

A little knowledge of computer systems and some programming would be enough to understand the content. This blog ought to be one of your first blogs on networking and its architecture. An interest in the topic will be sufficient to move forward.

Introduction

In this information age, where most, if not all the devices around us are getting connected, the internet stands as the most revolutionising technology used in our daily life, providing room for a great many opportunities.

The internet is a network of a large number of devices connected and sharing information, together forming the WWW. This technology lays down the path to various computer science areas and sub-disciplines like cloud computing, IoT, Cybersecurity, web development…

It all comes down to the basic knowledge of servers and clients and how the network works. In this blog, I shall be describing the simplest server-client communication based on sockets and how to get that public. So, let’s begin!

Servers and Clients

For those who don’t know, in the simplest case, a network connection between 2 devices generally involves a machine (server) which provides functionality (help) to the other device (client) in its operation. Typically, they communicate using requests and response.

Before moving ahead, Let me introduce the following terms:

Network Interface

Communication between your computer and the network is through a network interface. Generally present in the form of a network interface card(NIC), virtual interfaces can also be implemented using software. Loopback Interface (mentioned below) is an excellent example of this. Use ifconfig in Linux and ipconfig in Windows for getting the list and details of the interfaces on your device. Generally, wl/wlan represent wlan interface, eth/en represent ethernet and lo represent the loopback virtual interface.

Sockets and ports

OK! So we are ready for a connection, aren’t we?

Socket-port

Well, every connection has two ends. And in our case, one is with the client, and one is with the server. At both the end, network writes the received data packets to a file dedicated for that particular connection. Sockets are the file descriptor to the file, which is used to write the data.

Any application may be up at any time to communicate on the connection or establish new ones. Every program should naturally keep an eye on separate file descriptor on the same device. How to handle making separate files for each? and how would new requests from the same device or other be handled, so that they communicate with the exact application they want to? For this, we add a Port number to the network address of the device to identify them separately. Thus, we get a unique address:port combination to reach a particular address. It is like a set of (non-physical) channels where a process can dock to wait for new connections. This combination is called Socket address and is then used to identify separate sockets to identify the file to write data to. Evidently, the targeted application will be listening on the same socket. There are some standard port numbers which various server process take to listen to new requests. Have a look on this wiki.

Localhost

Just Localhost things!

Yo! this is where things get interesting! hmm…

Say your server is ready to be used. You got to check if it is working before deployment! Now Localhost is how you can do so.

As the name suggests, it is used to access network services running on the same device (thus fit for our use). What actually happens is that any request to the localhost isn’t forwarded to the external network interface, but to the Loopback interface. So, we can request any process listening to a socket on the loopback address from the same device itself! The addresses 127/8 (127.0.0.0 - 127.255.255.255) are bound to the loopback interface.

The word ‘localhost’ is mapped to 127.0.0.1 for you already! (Look at the /etc/hosts file in Linux). Just set up your server to listen to any port* on the localhost and try pinging it!!

Moving further, there’s another interesting address: 0.0.0.0. It is the joker IP address where we do not want to specify a particular IP address for our connection. If any data directed to the computer (by any IP address) through any interface (including loopback) is received, you can listen to it via the ‘joker’ address, on the particular port. Thus, it can be used as an IP address in the server instead of any particular IP address from any interface. This will enable you to connect to your server from any device in all the network the device is connected.

Ports 1–1024 are privileged so don’t use them (or get root permission).

Let’s get our hand dirty!

So, after much of the concepts, finally, it’s time to get to some code. I planned to provide the steps with their explanation! But I found resources like this helpful as well. Meanwhile, you can have a look at some code explanation here.

Server code

Client Code

Run the Server code in a terminal. The server will start waiting for a client request. Next, run the client code in a different terminal. If you had done it right, the server and the client must have exchanged their messages.

But broadly, the steps are as follows:

Make a socket object providing various options and types and protocols.
bind to an interface or all.
Start listening to the socket and respond to incoming requests.
Read and Write to the socket file.

Examples

vishwasmittal/bare-socket

Simple server and corresponding client written in bare sockets in multiple languages. The server should be written in…

Simple implementation of Socket based server-client programs in multiple languages.

dev-ritik/files_share

Web development proficiency. Contribute to dev-ritik/files_share development by creating an account on GitHub.

File sharing web and command-line app in C++ based on sockets.

Frameworks

Well, Socket level programming is not done generally in the production servers. These are abstracted and handled by various popular frameworks developers use to build and deploy a production-grade web application to handle real-time traffic from users. While developing simple web applications you may not even know what’s going at the socket level and how connections are handled. Having a framework comes with lots of benefits apart from abstraction. They may be fast, scalable, secure and tested, well documented, have good community support, etc. Some of the leading frameworks of 2020 can be found here.

Going Public

It’s quite cool and sometimes important to be able to access the server on your device from some other device. As such, it’s easy to do so. If you are not on a public-facing device (connected directly to the internet with public IP), you need to use some tricks to be available to anyone on the internet.

For the former case, directly use 0.0.0.0 or your public IP address as the IP address to host your server. For the latter case, we can achieve the same within the private network. Host the server on 0.0.0.0 or the assigned private IP address of the device on the network*. Now on the new device (on the same private network), use that same private IP:port to access the service.

To make your server public from a private network, have a look on how to do it through Port Forwarding. You may be required to take the help of your ISP here. If that looks complex, you can use services like ngrok or Serveo. They help to directly and simply do the task for you.

*Use ifconfig in Linux and ipconfig in Windows for getting the private IP address.

CLI Utilities

There are a bunch of useful utilities to test and interact with servers on the network. These include Ping, Curl, Wget, etc.

Curl:

$ curl localhost:8000
<HTML>
<Head>
<Title>Title: Bare Socket</Title>
</Head>
<Body>
 <center>
 <h1>
 Hello from server!
 </h1>
 </center>
</Body>
</HTML>

Wget: Save the received HTML text into a file.

$ wget localhost:8000
 — 2019–12–23 17:44:16 — http://localhost:8000/
Resolving localhost (localhost)… 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified
Saving to: ‘index.html’
index.html [ <=> ] 139 — .-KB/s in 0s
2019–12–23 17:44:16 (5.31 MB/s) — ‘index.html’ saved [139]

Being pretty simple in use, it’s good to try them all once.

🚀 Ready to Conquer the Digital Frontier? This glimpse is just the beginning of your journey into Computer Networking and Services. Dive into my next blog on ‘HTTP & WebSocket’ for an even more thrilling ride. The tech adventure continues! 🌐💻

HTTP & WebSockets

How are they connected to sockets? How is HTTP evolving? HTTP/2? HTTP/3? QUIC? Is WebSockets the same as Sockets?…

NpChat

2019-12-07T00:00:00+00:00

an NDN based multimedia sharing app

In a cyber universe dominated by giants like Google and Facebook, our everyday internet travels often lead to their massive servers. From E-commerce to Social Media and Web Streaming, it’s a giant corporation’s world, and we’re just browsing in it. But here’s the catch - our precious user data gets stored and played by their rules, often buried in the terms of service we hastily agree to.

But what if there’s another way? NpChat steps in as a practical, experimental alternative - a decentralized, end-to-end encrypted social multimedia app. No glorification, just a straightforward exploration into the possibilities of a more secure, user-centric design. Are you ready for a digital shift that puts control back in your hands? Let’s delve into the practicalities. 🚀🌐💻

NpChat (from NDN+ Snapchat) is an Android application that allows users to capture and share multimedia with friends in a secure and fully decentralised way, while still giving users complete control over their data. It is based on Named Data Network, a proposed evolution of the IP architecture that uses data-names(identifiers) as a kind of address and not just IP addresses. NDN aims to change the network fundamentally from end to end communication to content-based, wherein, not only IP address but also the data identifier and only the identifier can also be used to exchange information. More on NDN

NpChat

As NpChat is based on NDN, many of its concepts are blended into the app. As such, it may require further clarification of the idea.

As the Network layer is different in NDN and IP, it requires specially designed libraries and technologies to work with. NDN has a lot of support libraries written in different languages actively developed to support application development. As NpChat is developed for Android, it uses java library jNDN and some C++ libraries via the native interface. To connect to an NDN network, it uses NFD (an android app to use NDN over IP network). This daemon is required for NpChat to work and send/receive messages.

Let’s get into the working of the app

Initialising

The app on it’s initialising does the following stuff:

Ensure NFD is installed and running in the background
Initialize Psync library using JNI and native code. More on Psync (partial state synchronisation protocol for NDN can be found here.
Initialise the Psync partial producer. Register syncPrefix in NFD and sets internal filters for “sync” and “hello” under Prefix. Also, provide the prefixes available to synchronise
Initialise consumer manager and add consumer for each friend. Send a hello interest according to the Psync protocol and wait for the hello reply to synchronise further.
Make a new NDN Face
Make new AndroidSqlite3Pib object to help with SQLite for using PIB NDN to store public keys
Make new TpmBackEndFile object to store and access private keys and provide its location to AndroidSqlite3Pib object
Make a new KeyChain object to help provide a set of interfaces to the security library
Fetch/generate the pub key for the user namespace in the app using the keychain object and store the PIB key and related identity and certificates.
Set the signing information for the face using the keychain and the generated certificate for the name
Set up the custom memory cache object to help serve data to interests after it’s not in the cache (like after app restart).
Register names to NFD for receiving and responding to incoming interests for data. Interests may be of data, file, cert, friends, network-discovery, discover, etc. While others are for the general exchange of data and certificates, network-discovery is used to find out other instances of NpChat and publish our namespace address to them.
Start network discovery using NFD multicast discovery, NSD (DNS-SD), and send network-discovery get and establish named routes to other users.
Call processEvents on a separate network thread and loop it continuously.

With these things set up, the user can now interact with the app and send and receive media. But before doing that, he has to make friends.

Friends

Akin to the real world, making friends on NpChat also requires trust among the two parties. Trust in this context is the acceptance and verification of the signed data from the other party. This means that we accept the public key to be of that particular user. The acceptance of this public key involves various trust models as proposed and implemented in the app.

During the process of making friends (or building trust), they exchange their public keys. Once two users accept their friendship, they issue certificates to each other by signing the new friend’s key. The certificate is designed to certify the user to own a namespace (in this case, for the app).

So the main problem in a non-centralised app like this is to get suggested potential friends-list so that we can add new friends. Without a central list of available users, this is a rather challenging job. So, to discover and trust new users to be a new friend, the app uses:

Physical certificate exchange through QR exchange
Mutual Friend’s friend list: User can share his list of trusted friends with their friends so a user can have a list of potential friends
Domain: A user can easily trust and find other users from the same organisation, part of their namespace

Taking another look in the feed based sharing, it is evident that a user subscribes to friends feed. Once a friend publishes a piece of information, it is published to all his friend (who has subscribed to his feed). NpChat uses partial sync mode in PSync to realise this pub-sub functionality. Psync is designed to inform the consumers of a new piece of data produced. The friend can then request the producer (in the form of interest) to get that data back. Also, while logging into the app, a user can sync his state with that of the producer. This will get the latest data from him.

Access Control

A user in NpChat may share a piece of data with a particular user or a group of user he wants to. Unless we encrypt the data, every user on NDN can access it, given he knows its unique name. This possesses a significant challenge to privacy. So to allow only a certain number of people to access a particular data, NpChat uses both Asymmetric and Symmetric key encryption.

As mentioned above, NpChat user stores the public keys of his friend upon adding him as a friend. The data to be sent is encrypted using a symmetric key (content key). To send the data, NpChat sends SyncData (metadata) to all the friend through Psync. This SyncData is itself encrypted by the selected friend’s public key. Upon receiving this information, each recipient in the set of authorised friends fetches both the filename and the encrypted content key, decrypts the content key and then decrypts the data using the content key. Even if an unauthorised friend retrieves the encrypted data, he/she will not be able to decrypt the content key and thus cannot decrypt the data.

For more and detailed description of the app, head over to the reference paper and the code repository.

Reference

http://dl.acm.org/citation.cfm?doid=3357150.3357402

Way Ahead

As of now, NFD is available on the play store, and NpChat is almost ready to be released. The source code of the app is available under GNU GPL v3.0 here. The primary work is done, and the app can be used to share data in a local network. Some of the considerable work left, for the app to be done effectively are:

Improving the user interface
Connecting with the NDN Testbed
Managing Groups
Enhancing user discovery
Handling edge cases

among others. More information on this can be found here.

GSoC NDN 2019 Report

2019-08-25T00:00:00+00:00

Android App Improvement

Introduction

NpChat is a photo and file-sharing application built on Android and is inspired by Snapchat. It runs over the Named Data Network (NDN) and focuses on decentralised information sharing architecture. It stands as one of the best examples of Android application developed on the new Internet architecture.

The project description can be found here. During Google Summer of Code 2019, I contributed to the project and developed several features to improve the NpChat app, its supporting library(Jndn) and the Identity Manager.

Work

During the GSoC period, I worked on the following,

Replaces FaceProxy with Memorycache (#132) : Replaced FaceProxy (which used to store and serve data for interests) class with Memorycache class which uses MemoryContentCache (from the jNDN library).
Aimd pipelining in Segment fetcher (#24) : Updated SegmentFetcher.java from the jndn library to use AIMD pipe-lining in place of the fixed window fetching. Ran some tests on minindn to compare these strategies among the jNDN and ndn-cxx libraries, the most recent of the results can be found here. After a long review period and a list of changes, it was successfully merged.
Introduce MVVM architecture (#161) : The state of the code at that point was more or less ad-hoc with things like memory and network access specific code placed and used in the view itself against the encouraged application architecture patterns like MVVM. This Pull Request moved and reorganised the code to single Activity multiple Fragment MVVM architecture, with operation logics separated and organised into classes and packages.
Add location co-ordinates to pictures (#162) : Added a feature to let user geo-tag a picture. Once Geo-tagged, the location is shared by adding the co-ordinates into the Exif metadata and transformed to ground address using Geocoder class.
File sharing (#163) : Improved the pictures sharing and file browsing experience and separated them into status and files, to be viewed differently.
NDNCert java client (#1): For the users to have a namespace, they must have a certificate for it. Previously, a self-signed certificate was used to verify the ownership. This PR updates the Android Identity Manager to get signed certificates of the namespace using NDNCERT. This will be used to prove the ownership using the trust agent.

GSoC contribution

Here is a list of all contribution made during the GSoC period.

(#132) Replaces FaceProxy with Memorycache
(#134) Removes publishData code from FilesActivity
(#154) Replace Log with Timber
(#155) Add AboutActivity
(#161) Introduce MVVM architecture
(#162) Add location co-ordinates to pictures
(#163) Picture selection
(#24) Aimd pipelining in Segment fetcher
(#25) Fix null object error in AIMD SegmentFetcher
(#1) Ndncert java

These PRs spans accross NpChat, jndn and android-identity-manager

TODO

Publishing the app to Play Store.
Adding ECDH key encryption to android-identity-manager.

Challenges

As a part of the project, I faced some challenges, which gave me experiences in new fields

Moving to decentralised from the popular centralised server model.
The nature of the work (pretty much new and unprecedented).
Establishing and updating development environments and NDN dependencies.
Trying to establish ECDH key exchange.

Future Works

I plan to continue working on the NDN concept, developing and contributing to open-source application and utilities, in the days to come.