Late last week I upgraded my firewall to PFsense 2.3 and since then I have been having a daily issue that I don't even know how to begin to hunt down. At some random point, 12-16hrs, all traffic seems to get blocked. External traffic to the WAN interface, LAN traffic, everything. Today I managed to pull up the console on the fw itself and saw it was matching everything against the default 'deny all' rules on the interfaces. Traffic from the fw to any dst worked fine, though. A reboot cleared it up. But then it takes a bit to get the rest of my devices happy from being blocked forever.


I figure that my rules are generally OK as after a reboot all traffic gets handled normally. Despite that, I have added a 'allow all LAN ' to the WAN fw ruleset. We shall see if that helps. Any ideas? The only thing I can think of is that I do have a bit of an odd setup as it pertains to my wireless network. Its on its own interface on the fw, and all traffic is pushed through a VPN unless its LAN traffic. Then that is routed through the WAN GW. It works just fine, but that is the only thing I can think of that would fall under the default block.


 

ETA:


On second thought, I deleted that new rule. If traffic works fine now, its not a missing rule. Checked my logs and it explicitly denied my VPN connection and SSH connection, both of which are the first two allow rules in the chain.

Interesting.. I ran pfSense for a few years, then moved to Check Point GAIA. Never ran into that issue when I was running pfSense though.

Its the first day off I have had since the issue cropped up. Still nothing. I really want to catch it as soon as it happens so the logs are easier to sort though.

Have you tried the pfSense forums?  I haven't run it in a few years, but I recall the forums as being pretty helpful.

If anyone cares, I think I found the root cause. My VPN was setup via cron to get a HUP at the top of every hour, so that it was always cycling IPs. On the new upgrade, they made things a lot more modular. Two different packages were detecting the HUP and regenerating all the rulesets. When the stars aligned just right, both processes would attempt to rebuild at the same time crashing each other in a race condition. Hence only leaving the default deny rule. Ive bumped the cron to only once a day and have been issue free since.

Last time that happened to me it turned out to be a licensing issue.  The fw decided that any incoming packet to an address within any of the listed internal networks incremented against the license pool whether or not it was allowed traffic or a device was even there.



Someone was running an automated scanning tool on an outside network on a routine basis that coincided with the license depletion.



Maybe the NAT tables are maxing out and not clearing.  I know on the cisco stuff you could clear the tables with "clear xlate".  Next time it's locked up try "To clear all active connection states, visit Diagnostics > States, then go to Reset States tab."

 
I hit a bug when upgrading a while back where if you elected to do a backup automatically as part of the upgrade, the disk would get full and the upgrade would fail. Totally bricking the fw. So after I did a reinstall and imported my old config, things were going seemingly well. Until randomly sites, devices, etc would have just flat fail to work. Then magically they would start working again. Turns out my old backup config didn't have the mbufs bumped from 25k to 1M as is required with the Intel quad NICs in the box. Was just randomly bouncing off the ceiling until there was a free chunk for the request. Drove me nuts until I spotted it on the dashboard.