Thank you for reporting this! This is certainly a bug since after a project is made private (and hence is no longer considered open source) we do not seem to remove the “Free and Open Source” flag.
I cannot see any FOSS related option in advanced settings. The only options are:
JUnit support
GitHub Status updates
Permissive building of fork pull requests
Only build pull requests
Auto-cancel redundant builds
I’ve found “Free and Open Source” for other projects that are Open Source and switch is on. But for one that was private before, now public there no such switch.