FYI: @rockdaboot

Thanks for letting me know about this patch ! Great work !

I didn't even make a release of libhsts yet. Nor did I make a real implementation by myself (but it's on my list for wget and wget2 - just striked out curl ;-). I was so busy the last weeks with polishing the GnuTLS build architecture... But I take this as a butt-kick to get busy with libhsts again :-)

The test 1140 failure is irrelevant and due to commit b6a16af, subsequently fixed in commit 810ce31

@zagor let me know if you need any help getting changes to the test scripts done so we can have a test or two of this added.

@bagder @vszakats Thanks for the reviews, fixes are coming.

@bagder Sure, if you can point to examples of tests that use similar features that would be a good help

I think the PSL test, test 1136, is a good example. Note how it sets "PSL" as a required feature for the test to run.

The runtests.pl script first checks if the build has the feature enabled and if so, sets the associated variable. The logic that then makes sure that the test's required feature matches what the local build features is here.

You could use a similar approach for HSTS.

I would also suggest that you make the test depend on "debug", which makes it only work on debug builds but then you can add custom logic #ifdefed on DEBUGBUILD that isn't present in release builds. It could for example include something like:

#ifdef DEBUGBUILD
char *special = curl_getenv("CURL_HSTSFILE");
if(special) {
  /* use this file name instead of the real one */
  free(special);
}
#endif

You can then make the test case set that environment variable. Test 1136 sets the TZ environment variable, which shows how to do it.

@zagor you up to getting this polished and ready for merge?

I'll volunteer to write up a travis adjustment for it post-merge.

Yes, sorry for the delay. I had a long vacation and sort of forgot about this. I'll trim the final bits soon!

Fixed the review comments. Test case coming soon.

I think the tests look fine!

Some questions/wishes on the patch in general:

1. The 'hsts' handle should probably be moved to the UrlState instead of the UserDefined in lib/urldata.h for consistency. The UserDefined tend to hold the exact values set with curl_easy_setopt() while the UrlState is for holding things that are updated run-time in the handle.
2. I would actually presume that (some) users will want to specify a dedicated HSTS file when doing transfers, rather than just having the default path set at build-time. To support a custom path, the loading of the HSTS file can't be done in Curl_open() since that happens before any options is set in the handle.
3. I would like users to be able to switch off the HSTS use in case they want to test stuff without it (a little in the --insecure spirit). If we do (2) that way could then be to just point to a missing/other file.

Doesn't #1 and #2 combined mean that I'd have to initialize libhsts once for every easy handle?

Initialization is not very lightweight, as the default database is ~700KB

Sorry, forgot to weigh in on this but thanks for underscoring the issue for me. Here's what I think...

Loading 700KB only to do a single transfer, kill the handle and then do another one that loads the same 700KB again is certainly almost like a nightmare scenario to many users. At the same time, doing it unconditionally with a fixed path in curl_global_init() because it was enabled at built-time is also inconveniently inflexible and will also cause sad faces (possibly in another group of users).

However, doing HSTS globally still has so many downsides that I can't see it being a very good way forward. For example users will be stuck with that single HSTS file, no matter how many easy handles or threads it uses.

I think we pretty much have to do the load per-handle in spite of all. We should probably then only do it for users that ask for it, and then we can allow a custom path to the file.

As a follow-up, we can then add HSTS sharing to the share API so that a user can select to share that single HSTS between exactly those handles they think is suitable. An application could then even create/kill easy handles and re-use the same cached share object if they want to use HSTS without reloading it many times.

Initialising the HSTS object on first-use and sharing it afterwards looks like a nice solution — hsts_search() doesn't seem to want to modify the HSTS object, which helps.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Thanks for the reminder, bot. I haven't abandoned this, just being really bad at finishing what I started. 😄

I have @zagor's work rebased and squashed in a separate branch.

I have yet again rebased this work in my separate branch. There is an interest in HSTS from users and I'm thinking of how to push this forward. @zagor, are you still interested in working on this?

I'm very embarrassed but yes, I would still like to see this go in. I'll try to make time for it soon.

I think we should presume users of this would like to use their own crafted preload file (simply because the full one is too big) and/or that we should populate a file/cache at run-time based on incoming headers. Possibly we could make the preloaded list get passed to libcurl via a pointer/buffer somehow so that applications could have a built-in set.

I suggest we split runtime caching into a separate PR. This one has been open for much too long already.
I agree with adding a setopt() for the dafsa file. Do you think the configure option should be removed entirely, or should it be kept as an optional fallback to be used if no file was given with setopt()?

Maybe we want to specify a default one for the curl tool?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.