The lower (with the serial number) is the "weapon". An upper is a collection of parts. You may collect and possess as many uppers as you like and swap them to your hearts content. There is no law that prohibits the importation of uppers into California, so I can't provide you a law that says it's OK, they don't usually write laws that way. You may swap the upper for a flat top with no issues.
I know those items are illegal to purchase now for the most part, but not sure if they are ok to 'replace.'
This is not correct. As long as you are not replacing your lower receiver, you may remove and replace parts freely.
The only thing you should be careful of is barrel length. Federal law classifies any rifle with a barrel under 16" as a Short Barreled Rifle (SBR). To possess one, you have to register it with the ATF in accordance with the National Firearms Act (NFA) AND it must be permissible in your state. California has a permit process for these but they refuse to issue any permits. Hence, you cannot possess a SBR in CA unless you fall under one of the exceptions (LE Agency, military, some movie permits, etc.).