Based on how the Japanese command and the decisions they would have likely made, there would have been almost no impact on the war. The Iowa class battleships, which were awesome, but arguably had little impact strategically would have had their resources shifted to speed up the building of the Essex class carriers, with little expense on the Soviet Union or the British. The US would not need to defend California or even shift resources for it-the US can easily just activate their militia (remember the Second Amendment). Japan would not have invaded (or even thought of) invading the continental US, and capturing Midway or Hawaii (a formidable task in itself) would still have made it hard, if not impossible, to win in an invasion. Japan would have needed more ships and men than planned for Operation Downfall, which would be impossible, considering only the US could muster that large of a force. Japan would also have to deal with China, the threat of the Soviet Union, and the British. It would have been a waste of a victory, as Japanese troops would probably have been slaughtered on the beaches of Midway.
Now if the Japanese political and military leadership was headed by people who know what they’re doing, then absolutely! This would have set the US back quite a lot. The Japanese then should have adopted a strong dug in defence, operating the capital ships only in range of powerful land based aircraft and submarines, coming up with an effective convoy system, and working with the army to coordinate operations. This could have taken up so many US resources to beat that the Soviets would have had a lesser victory at Stalingrad, and Rommel could have held out longer in Tunisia if there were less forces in Operation Torch.
The best thing for the Japanese to do, of course, is to attack the Soviet Union. The Far Eastern troops were grossly exaggerated by the Soviets (which the Japanese willingly believed after Kahlkin Gol), and replacements for the ones that have been siphoned off west were generally worse, with little training. The Japanese air force was less outnumbered by the Soviets than the Germans were, and had even more of a quality advantage. The Japanese could have easily destroyed most of the Soviet Pacific fleet in days, destroyed the Red air force in an even more total fashion than the Germans did, and captured Vladivostok. Japan’s decisions are kind of stupid when they basically thought that they had a greater chance of defeating the US, a nation that has had their resources not yet tapped in any way (except Lend Lease) than the Soviet Union, whose cream of forces were destroyed in 1941, their replacements almost always understrength (most Soviet cavalry divisions were less than a third of a real division, rifle and tank brigades were only a quarter of even a small World War 2 division, and rifle divisions were only half as effective as a German infantry division), suffered enormous equipment and officer shortages, and had horrible logistics. In addition, many Soviet units had little training, and were expected to get that on the job fighting, leading to huge casualties. And that’s the west! The Far East forces were infinitely more inferior. As for oil, Japan could have reduced American unhappiness by doing withdrawing partially from French Indochina and reducing operation in China, both of which are necessary anyway for mustering forces. Diplomatic decisions are much easier to reverse than military ones, and the US would have saw the invasion as a removal of a threat to American interests in the Pacific (both Japan and the Soviet Union would have their hands tied up). Coupled with a better German strategy, the war in the East could have been won in 1941.
I strongly recommend you read Nigel Askley’s books, who give the most detailed account of Operation Barbarossa, detailing German and Soviet mobilisations, production values, and reinforcements, all used for a war simulation. I haven’t bought them all and don’t read them entirely (only the parts I’m interested in), but they are very good books, and more are being written. The part about Soviet Far East forces (Volume 3B) is very well done, and they also cover the Soviet and German air forces and navies.