Windows 10 & 11 have already have a firewall. Its settings are hidden somehow and difficult to configure but it is there. You can configure the Windows firewall using Binisoft‘s (now owned by Malwarebytes) Windows Firewall Control.
But there is a much better option for you: Add a second layer of Windows protection, using TinyWall, a free and lightweight complete firewall.
Tinywall started in 2011 as a simple interface for Windows Firewall tuning. Version 3 is now a standalone firewall, that can be run without or side-by-side with Windows own built-in firewall.
Tinywall offers some very interesting features, like BOOT TIME filtering, block of all outgoing connections until you allow them manually, whitelist of programs, compatible with Windows Updates and much more.
TinyWall runs silently with no popups or warnings of anykind. You have to manually check what is blocked (Show Connections) and allow the connections permanently or for a while.
TinyWall works also with older Windows versions (7, 8 or 8.1) but this will be dropped soon.